Jul 18 16:34:39.844228 ip-10-0-142-77 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 18 16:34:39.844242 ip-10-0-142-77 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 18 16:34:39.844250 ip-10-0-142-77 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 18 16:34:39.844553 ip-10-0-142-77 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 18 16:34:50.031367 ip-10-0-142-77 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 18 16:34:50.031389 ip-10-0-142-77 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7c1f996bb58c43d6bbaaab7beb4999f8 --
Jul 18 16:36:21.192107 ip-10-0-142-77 systemd[1]: Starting Kubernetes Kubelet...
Jul 18 16:36:21.584593 ip-10-0-142-77 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 18 16:36:21.584593 ip-10-0-142-77 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 18 16:36:21.584593 ip-10-0-142-77 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 18 16:36:21.584593 ip-10-0-142-77 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 18 16:36:21.584593 ip-10-0-142-77 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 18 16:36:21.585566 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.585461    2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 18 16:36:21.590015 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.589999    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 16:36:21.590015 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590016    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590020    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590023    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590027    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590030    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590032    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590035    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590039    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590043    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590046    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590048    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590051    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590064    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590067    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590070    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590072    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590075    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590077    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590080    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590083    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 16:36:21.590078 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590085    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590088    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590091    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590094    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590096    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590099    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590101    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590104    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590106    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590109    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590111    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590113    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590116    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590119    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590121    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590123    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590126    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590128    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590131    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 16:36:21.590550 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590133    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590135    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590138    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590140    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590143    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590145    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590148    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590150    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590153    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590157    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590160    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590163    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590166    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590169    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590171    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590175    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590178    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590181    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590184    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590186    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 16:36:21.591027 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590189    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590192    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590194    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590196    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590199    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590202    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590204    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590206    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590209    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590212    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590214    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590218    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590221    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590223    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590226    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590229    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590231    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590234    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590236    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 16:36:21.591795 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590239    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590241    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590244    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590248    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590250    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590253    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590255    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590258    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590261    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590263    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590266    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590268    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590271    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590273    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590276    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590278    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590281    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590283    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590286    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.590288    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 16:36:21.592515 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591792    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591812    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591818    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591823    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591827    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591833    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591837    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591841    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591868    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591874    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591883    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591888    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591894    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591898    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591902    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591906    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591911    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591915    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591919    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591923    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 16:36:21.593062 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591927    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591931    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591935    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591944    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591949    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591952    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591956    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591963    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591969    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591974    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591978    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591982    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591987    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.591999    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592003    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592012    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592017    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592021    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592025    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592029    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 16:36:21.593580 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592034    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592038    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592042    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592046    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592051    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592055    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592060    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592070    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592074    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592078    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592083    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592087    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592091    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592095    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592099    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592103    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592108    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592112    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592115    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592119    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 16:36:21.594086 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592128    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592132    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592137    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592141    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592145    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592152    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592157    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592161    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592165    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592169    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592173    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592177    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592186    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592190    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592196    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592202    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592207    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592211    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592215    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 16:36:21.594571 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592219    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592223    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592231    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592237    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592241    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592251    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592255    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592260    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592264    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592268    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592272    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592276    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592280    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592319    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592335    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592340    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592345    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592349    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592352    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 16:36:21.595077 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.592356    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593069    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593079    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593082    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593086    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593089    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593092    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593095    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593098    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593100    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593103    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593105    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593107    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593110    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593114    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593118    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593121    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593124    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593126    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 16:36:21.595542 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593130    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593132    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593135    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593139    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593143    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593145    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593148    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593151    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593154    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593156    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593159    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593161    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593164    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593166    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593169    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593172    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593174    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593177    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593180    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593183    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 16:36:21.596009 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593185    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593188    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593190    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593193    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593195    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593198    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593200    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593202    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593205    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593208    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593211    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593213    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593216    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593218    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593221    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593223    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593226    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593228    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593231    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593233    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 16:36:21.596495 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593236    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593239    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593241    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593244    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593246    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593249    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593251    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593254    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593257    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593259    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593261    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593264    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593266    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593269    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593271    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593274    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593276    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593279    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593283    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593286    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 16:36:21.597104 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593289    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593292    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593295    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593297    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593300    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593303    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593306    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593309    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593312    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593315    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593317    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593320    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593323    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593326    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593328    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593331    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593333    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593335    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593338    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593341    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 16:36:21.597583 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.593343    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593408    2554 flags.go:64] FLAG: --address="0.0.0.0"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593415    2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593421    2554 flags.go:64] FLAG: --anonymous-auth="true"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593426    2554 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593430    2554 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593433    2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593437    2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593442    2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593445    2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593447    2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593451    2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593454    2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593458    2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593461    2554 flags.go:64] FLAG: --cgroup-root=""
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593464    2554 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593467    2554 flags.go:64] FLAG: --client-ca-file=""
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593470    2554 flags.go:64] FLAG: --cloud-provider="external"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593473    2554 flags.go:64] FLAG: --cluster-dns="[]"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593477    2554 flags.go:64] FLAG: --cluster-domain=""
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593480    2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593483    2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593486    2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593489    2554 flags.go:64] FLAG: --container-log-max-files="5"
Jul 18 16:36:21.598125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593498    2554 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593501    2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593504    2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593507    2554 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593510    2554 flags.go:64] FLAG: --contention-profiling="false"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593513    2554 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593527    2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593530    2554 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593533    2554 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593537    2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593540    2554 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593543    2554 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593546    2554 flags.go:64] FLAG: --enable-load-reader="false"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593549    2554 flags.go:64] FLAG: --enable-server="true"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593552    2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593556    2554 flags.go:64] FLAG: --event-burst="100"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593559    2554 flags.go:64] FLAG: --event-qps="50"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593562    2554 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593564    2554 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593567    2554 flags.go:64] FLAG: --eviction-hard=""
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593571    2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593574    2554 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593576    2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593579    2554 flags.go:64] FLAG: --eviction-soft=""
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593582    2554 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 18 16:36:21.598687 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593585    2554 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593588    2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593591    2554 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593594    2554 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593597    2554 flags.go:64] FLAG: --fail-swap-on="true"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593600    2554 flags.go:64] FLAG: --feature-gates=""
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593603    2554 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593606    2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593609    2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593612    2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593615    2554 flags.go:64] FLAG: --healthz-port="10248"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593618    2554 flags.go:64] FLAG: --help="false"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593621    2554 flags.go:64] FLAG: --hostname-override="ip-10-0-142-77.ec2.internal"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593624    2554 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593627    2554 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593630    2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593633    2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593637    2554 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593640    2554 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593642    2554 flags.go:64] FLAG: --image-service-endpoint=""
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593645    2554 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593648    2554 flags.go:64] FLAG: --kube-api-burst="100"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593651    2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593654    2554 flags.go:64] FLAG: --kube-api-qps="50"
Jul 18 16:36:21.599328 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593657    2554 flags.go:64] FLAG: --kube-reserved=""
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593660    2554 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593662    2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593665    2554 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593668    2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593670    2554 flags.go:64] FLAG: --lock-file=""
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593673    2554 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593676    2554 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593679    2554 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593684    2554 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593687    2554 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593690    2554 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593693    2554 flags.go:64] FLAG: --logging-format="text"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593696    2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593699    2554 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593704    2554 flags.go:64] FLAG: --manifest-url=""
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593707    2554 flags.go:64] FLAG: --manifest-url-header=""
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593712    2554 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593715    2554 flags.go:64] FLAG: --max-open-files="1000000"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593719    2554 flags.go:64] FLAG: --max-pods="110"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593722    2554 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593725    2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593727    2554 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593730    2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593733    2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 18 16:36:21.599921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593736    2554 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593739    2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593745    2554 flags.go:64] FLAG: --node-status-max-images="50"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593748    2554 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593751    2554 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593754    2554 flags.go:64] FLAG: --pod-cidr=""
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593756    2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593761    2554 flags.go:64] FLAG: --pod-manifest-path=""
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593764    2554 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593767    2554 flags.go:64] FLAG: --pods-per-core="0"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593769    2554 flags.go:64] FLAG: --port="10250"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593772    2554 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593775    2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01469a32cb14063db"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593778    2554 flags.go:64] FLAG: --qos-reserved=""
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593781    2554 flags.go:64] FLAG: --read-only-port="10255"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593783    2554 flags.go:64] FLAG: --register-node="true"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593786    2554 flags.go:64] FLAG: --register-with-taints=""
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593790    2554 flags.go:64] FLAG: --registry-burst="10"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593794    2554 flags.go:64] FLAG: --registry-qps="5"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593797    2554 flags.go:64] FLAG: --reserved-cpus=""
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593799    2554 flags.go:64] FLAG: --reserved-memory=""
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593803    2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593807    2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593810    2554 flags.go:64] FLAG: --rotate-certificates="false"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593813    2554 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 18 16:36:21.600535 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593816    2554 flags.go:64] FLAG: --runonce="false"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593819    2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593822    2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593825    2554 flags.go:64] FLAG: --seccomp-default="false"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593827    2554 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593830    2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593833    2554 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593836    2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593839    2554 flags.go:64] FLAG: --storage-driver-password="root"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593853    2554 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593857    2554 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593859    2554 flags.go:64] FLAG: --storage-driver-user="root"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593862    2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593865    2554 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593868    2554 flags.go:64] FLAG: --system-cgroups=""
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593871    2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593876    2554 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593879    2554 flags.go:64] FLAG: --tls-cert-file=""
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593881    2554 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593886    2554 flags.go:64] FLAG: --tls-min-version=""
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593889    2554 flags.go:64] FLAG: --tls-private-key-file=""
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593891    2554 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593894    2554 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593898    2554 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593901    2554 flags.go:64] FLAG: --v="2"
Jul 18 16:36:21.601186 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593905    2554 flags.go:64] FLAG: --version="false"
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593909    2554 flags.go:64] FLAG: --vmodule=""
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593913    2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.593916    2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594034    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594038    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594041    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594045    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594047    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594050    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594053    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594056    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594060    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594063    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594065    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594068    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594071    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594073    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594076    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594079    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 16:36:21.601812 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594082    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594084    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594087    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594089    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594092    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594095    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594097    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594100    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594102    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594105    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594109    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594111    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594114    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594116    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594119    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594121    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594125    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594129    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594132    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594135    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 16:36:21.602314 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594137    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594140    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594143    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594145    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594148    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594150    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594152    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594155    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594157    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594160    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594162    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594165    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594167    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594169    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594172    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594174    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594177    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594179    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594182    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594184    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 16:36:21.602813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594186    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594189    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594193    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594195    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594198    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594200    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594202    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594206    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594212    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594214    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594217    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594219    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594222    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594224    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594227    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594229    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594232    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594234    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594237    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594239    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 16:36:21.603329 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594242    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594244    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594247    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594249    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594252    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594254    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594257    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594259    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594261    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594264    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594267    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594271    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594274    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594277    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594281    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594284    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594286    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594289    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594292    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 16:36:21.603810 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594294    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 16:36:21.604298 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594299    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 16:36:21.604298 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594302    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 16:36:21.604298 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.594304    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 16:36:21.604298 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.595856    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 18 16:36:21.604298 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.602489    2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 18 16:36:21.604298 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.602505    2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 18 16:36:21.604298 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.602518    2554 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 18 16:36:21.604298 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.602524    2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 18 16:36:21.604298 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602577    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 16:36:21.604298 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602581    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 16:36:21.604298 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602584    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 16:36:21.604298 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602587    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 16:36:21.604298 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602590    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 16:36:21.604298 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602593    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 16:36:21.604298 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602597    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602599    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602602    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602604    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602607    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602610    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602623    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602627    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602630    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602632    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602636    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602638    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602640    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602643    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602646    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602648    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602651    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602655    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602659    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 16:36:21.604673 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602662    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602664    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602667    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602670    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602674    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602677    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602679    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602682    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602686    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602690    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602693    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602696    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602699    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602701    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602704    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602706    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602708    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602711    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602714    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602716    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 16:36:21.605174 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602718    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602721    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602723    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602726    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602728    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602731    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602733    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602736    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602738    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602741    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602743    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602746    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602748    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602751    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602753    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602756    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602759    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602763    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602765    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602768    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 16:36:21.605670 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602770    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602773    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602775    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602779    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602782    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602785    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602787    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602790    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602792    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602795    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602798    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602801    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602803    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602806    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602808    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602811    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602814    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602817    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602820    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 16:36:21.606167 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602823    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 16:36:21.606658 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602826    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 16:36:21.606658 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602829    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 16:36:21.606658 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602831    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 16:36:21.606658 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602834    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 16:36:21.606658 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602836    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 16:36:21.606658 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602839    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 16:36:21.606658 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602855    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 16:36:21.606658 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602858    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 16:36:21.606658 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602861    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 16:36:21.606658 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602864    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 16:36:21.606658 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602867    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 16:36:21.606658 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602870    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 16:36:21.606658 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602872    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 16:36:21.606658 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602875    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 16:36:21.606658 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.602880    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602991    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602994    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.602998    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603001    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603004    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603007    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603009    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603012    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603015    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603017    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603020    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603023    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603025    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603028    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603030    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603033    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603035    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603038    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603040    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 18 16:36:21.607045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603043    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603045    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603047    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603050    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603052    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603055    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603058    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603060    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603063    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603066    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603068    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603070    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603074    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603077    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603080    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603082    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603085    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603088    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603090    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 18 16:36:21.607516 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603092    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603095    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603097    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603101    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603104    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603106    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603109    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603111    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603114    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603117    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603120    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603122    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603125    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603128    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603130    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603133    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603135    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603138    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603140    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603143    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 18 16:36:21.607984 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603146    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603149    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603151    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603154    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603157    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603160    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603162    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603164    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603167    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603169    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603172    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603174    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603176    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603179    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603181    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603184    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603186    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603189    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603191    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603193    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 18 16:36:21.608482 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603196    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603198    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603200    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603203    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603207    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603210    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603213    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603215    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603218    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603220    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603222    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603225    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603227    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603230    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603232    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603235    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603238    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603240    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603242    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603245    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 18 16:36:21.609004 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:21.603247    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 18 16:36:21.609497 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.603251    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 18 16:36:21.609497 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.603357    2554 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 18 16:36:21.609497 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.605208    2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 18 16:36:21.609497 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.606347    2554 server.go:1019] "Starting client certificate rotation"
Jul 18 16:36:21.609497 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.606443    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 18 16:36:21.609497 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.606946    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 18 16:36:21.609497 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.608561    2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 18 16:36:21.612583 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.612567    2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 18 16:36:21.621806 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.621789    2554 log.go:25] "Validated CRI v1 runtime API"
Jul 18 16:36:21.623550 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.623534    2554 log.go:25] "Validated CRI v1 image API"
Jul 18 16:36:21.626538 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.626518    2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 18 16:36:21.628644 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.628622    2554 fs.go:135] Filesystem UUIDs: map[1715286b-7a70-4046-ace6-344d646f9165:/dev/nvme0n1p3 1b709d0e-ba4a-42f5-afd7-623dad32b4da:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 18 16:36:21.628724 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.628644    2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 18 16:36:21.632377 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.632359    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 18 16:36:21.635945 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.635825    2554 manager.go:216] Machine: {Timestamp:2026-07-18 16:36:21.634507081 +0000 UTC m=+0.332337755 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109584 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26584abb9aae48f2a96f2ca0e987ec SystemUUID:ec26584a-bb9a-ae48-f2a9-6f2ca0e987ec BootID:7c1f996b-b58c-43d6-bbaa-ab7beb4999f8 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:48:3d:d8:bc:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:48:3d:d8:bc:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:b2:f9:41:4d:5d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 18 16:36:21.635945 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.635939    2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 18 16:36:21.636063 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.636010    2554 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 18 16:36:21.638705 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.638671    2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 18 16:36:21.638877 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.638708    2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-77.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 18 16:36:21.639411 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.639399    2554 topology_manager.go:138] "Creating topology manager with none policy"
Jul 18 16:36:21.639464 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.639418    2554 container_manager_linux.go:309] "Creating device plugin manager"
Jul 18 16:36:21.639464 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.639428    2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 18 16:36:21.640051 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.640039    2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 18 16:36:21.641114 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.641102    2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 18 16:36:21.643564 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.643549    2554 state_mem.go:36] "Initialized new in-memory state store"
Jul 18 16:36:21.643697 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.643686    2554 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 18 16:36:21.643759 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.643723    2554 kubelet.go:486] "Attempting to sync node with API server"
Jul 18 16:36:21.643759 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.643737    2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 18 16:36:21.643759 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.643757    2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 18 16:36:21.643934 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.643771    2554 kubelet.go:398] "Adding apiserver pod source"
Jul 18 16:36:21.643934 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.643784    2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 18 16:36:21.644179 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.644164    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 18 16:36:21.644246 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.644187    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 18 16:36:21.646079 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.646061    2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 18 16:36:21.647575 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.647562    2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 18 16:36:21.647633 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.647591    2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 18 16:36:21.648525 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.648514    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 18 16:36:21.648570 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.648530    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 18 16:36:21.648570 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.648537    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 18 16:36:21.648570 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.648542    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 18 16:36:21.648570 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.648550    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 18 16:36:21.648570 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.648557    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 18 16:36:21.648570 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.648564    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 18 16:36:21.648570 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.648570    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 18 16:36:21.648754 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.648575    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 18 16:36:21.648754 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.648580    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 18 16:36:21.648754 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.648594    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 18 16:36:21.648754 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.648600    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 18 16:36:21.649931 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.649919    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 18 16:36:21.649967 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.649933    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 18 16:36:21.653211 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.653198    2554 server.go:1268] "Started kubelet"
Jul 18 16:36:21.653313 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.653286    2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 18 16:36:21.653518 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.653291    2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 18 16:36:21.653518 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.653429    2554 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 18 16:36:21.654071 ip-10-0-142-77 systemd[1]: Started Kubernetes Kubelet.
Jul 18 16:36:21.654778 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.654733    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l8h8w"
Jul 18 16:36:21.655355 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.655333    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-77.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 18 16:36:21.655438 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.655374    2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-77.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 18 16:36:21.655493 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.655476    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 18 16:36:21.655493 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.655487    2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 18 16:36:21.656313 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.656295    2554 server.go:310] "Adding debug handlers to kubelet server"
Jul 18 16:36:21.658340 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.658323    2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 18 16:36:21.659862 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.659831    2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 18 16:36:21.659948 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.659865    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 18 16:36:21.660045 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.660026    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-77.ec2.internal\" not found"
Jul 18 16:36:21.660959 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.660940    2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 18 16:36:21.661037 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.660958    2554 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 18 16:36:21.661037 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.660974    2554 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 18 16:36:21.661127 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.661055    2554 reconstruct.go:97] "Volume reconstruction finished"
Jul 18 16:36:21.661127 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.661084    2554 reconciler.go:29] "Reconciler: start to sync state"
Jul 18 16:36:21.661184 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.661169    2554 factory.go:153] Registering CRI-O factory
Jul 18 16:36:21.661231 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.661210    2554 factory.go:223] Registration of the crio container factory successfully
Jul 18 16:36:21.661318 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.661258    2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 18 16:36:21.661318 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.661267    2554 factory.go:55] Registering systemd factory
Jul 18 16:36:21.661318 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.661275    2554 factory.go:223] Registration of the systemd container factory successfully
Jul 18 16:36:21.661318 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.661294    2554 factory.go:103] Registering Raw factory
Jul 18 16:36:21.661318 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.661309    2554 manager.go:1196] Started watching for new ooms in manager
Jul 18 16:36:21.661649 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.661636    2554 manager.go:349] Starting recovery of all containers
Jul 18 16:36:21.663003 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.662952    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l8h8w"
Jul 18 16:36:21.663751 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.663721    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-77.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 18 16:36:21.663910 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.663881    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 18 16:36:21.665190 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.663996    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-77.ec2.internal.18c36f995147893d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-77.ec2.internal,UID:ip-10-0-142-77.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-77.ec2.internal,},FirstTimestamp:2026-07-18 16:36:21.653170493 +0000 UTC m=+0.351001168,LastTimestamp:2026-07-18 16:36:21.653170493 +0000 UTC m=+0.351001168,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-77.ec2.internal,}"
Jul 18 16:36:21.674413 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.674393    2554 manager.go:354] Recovery completed
Jul 18 16:36:21.679140 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.679128    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 16:36:21.680882 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.680866    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 16:36:21.680963 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.680898    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 16:36:21.680963 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.680914    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasSufficientPID"
Jul 18 16:36:21.681384 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.681367    2554 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 18 16:36:21.681452 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.681384    2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 18 16:36:21.681452 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.681402    2554 state_mem.go:36] "Initialized new in-memory state store"
Jul 18 16:36:21.681452 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.681405    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 16:36:21.682258 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.682246    2554 policy_none.go:49] "None policy: Start"
Jul 18 16:36:21.682322 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.682264    2554 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 18 16:36:21.682322 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.682275    2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 18 16:36:21.682322 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.682245    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 16:36:21.682439 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.682345    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 16:36:21.682439 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.682364    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasSufficientPID"
Jul 18 16:36:21.683103 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.683090    2554 policy_none.go:47] "Start"
Jul 18 16:36:21.718413 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.718399    2554 manager.go:337] "Starting Device Plugin manager"
Jul 18 16:36:21.730465 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.718426    2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 18 16:36:21.730465 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.718435    2554 server.go:87] "Starting device plugin registration server"
Jul 18 16:36:21.730465 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.718701    2554 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 18 16:36:21.730465 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.718712    2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 18 16:36:21.730465 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.718909    2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 18 16:36:21.730465 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.718980    2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 18 16:36:21.730465 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.718989    2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 18 16:36:21.730465 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.719581    2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 18 16:36:21.730465 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.719632    2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-77.ec2.internal\" not found"
Jul 18 16:36:21.758156 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.758131    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 18 16:36:21.759545 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.759528    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 18 16:36:21.759616 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.759563    2554 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 18 16:36:21.759616 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.759581    2554 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 18 16:36:21.759616 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.759612    2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 18 16:36:21.761970 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.761950    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 16:36:21.818942 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.818921    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 16:36:21.823498 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.823483    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 16:36:21.823595 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.823515    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 16:36:21.823595 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.823539    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasSufficientPID"
Jul 18 16:36:21.823595 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.823576    2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-77.ec2.internal"
Jul 18 16:36:21.836574 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.836527    2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-77.ec2.internal"
Jul 18 16:36:21.836574 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.836549    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-77.ec2.internal\": node \"ip-10-0-142-77.ec2.internal\" not found"
Jul 18 16:36:21.850334 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.850315    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-77.ec2.internal\" not found"
Jul 18 16:36:21.860378 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.860352    2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal"]
Jul 18 16:36:21.860422 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.860416    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 16:36:21.861861 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.861829    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d5d6ad110b232b65a608991bf5f07dc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal\" (UID: \"d5d6ad110b232b65a608991bf5f07dc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal"
Jul 18 16:36:21.861934 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.861880    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5d6ad110b232b65a608991bf5f07dc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal\" (UID: \"d5d6ad110b232b65a608991bf5f07dc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal"
Jul 18 16:36:21.862047 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.862036    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 16:36:21.862084 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.862058    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 16:36:21.862084 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.862072    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasSufficientPID"
Jul 18 16:36:21.863080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.863068    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 16:36:21.863195 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.863173    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal"
Jul 18 16:36:21.863229 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.863200    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 16:36:21.863744 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.863727    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 16:36:21.863798 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.863733    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 16:36:21.863798 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.863768    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 16:36:21.863798 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.863777    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasSufficientPID"
Jul 18 16:36:21.863952 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.863753    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 16:36:21.863952 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.863882    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasSufficientPID"
Jul 18 16:36:21.865500 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.865486    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal"
Jul 18 16:36:21.865551 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.865511    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 18 16:36:21.866203 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.866186    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasSufficientMemory"
Jul 18 16:36:21.866269 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.866216    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasNoDiskPressure"
Jul 18 16:36:21.866269 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.866230    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeHasSufficientPID"
Jul 18 16:36:21.889356 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.889337    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-77.ec2.internal\" not found" node="ip-10-0-142-77.ec2.internal"
Jul 18 16:36:21.893664 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.893649    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-77.ec2.internal\" not found" node="ip-10-0-142-77.ec2.internal"
Jul 18 16:36:21.950591 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:21.950563    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-77.ec2.internal\" not found"
Jul 18 16:36:21.962156 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.962132    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d5d6ad110b232b65a608991bf5f07dc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal\" (UID: \"d5d6ad110b232b65a608991bf5f07dc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal"
Jul 18 16:36:21.962211 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.962162    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5d6ad110b232b65a608991bf5f07dc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal\" (UID: \"d5d6ad110b232b65a608991bf5f07dc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal"
Jul 18 16:36:21.962211 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.962183    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a00913a0b98636dd43acae671203e933-config\") pod \"kube-apiserver-proxy-ip-10-0-142-77.ec2.internal\" (UID: \"a00913a0b98636dd43acae671203e933\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal"
Jul 18 16:36:21.962272 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.962232    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d5d6ad110b232b65a608991bf5f07dc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal\" (UID: \"d5d6ad110b232b65a608991bf5f07dc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal"
Jul 18 16:36:21.962306 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:21.962267    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5d6ad110b232b65a608991bf5f07dc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal\" (UID: \"d5d6ad110b232b65a608991bf5f07dc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal"
Jul 18 16:36:22.050793 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:22.050764    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-77.ec2.internal\" not found"
Jul 18 16:36:22.063133 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:22.063106    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a00913a0b98636dd43acae671203e933-config\") pod \"kube-apiserver-proxy-ip-10-0-142-77.ec2.internal\" (UID: \"a00913a0b98636dd43acae671203e933\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal"
Jul 18 16:36:22.063192 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:22.063169    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a00913a0b98636dd43acae671203e933-config\") pod \"kube-apiserver-proxy-ip-10-0-142-77.ec2.internal\" (UID: \"a00913a0b98636dd43acae671203e933\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal"
Jul 18 16:36:22.151627 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:22.151553    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-77.ec2.internal\" not found"
Jul 18 16:36:22.192095 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:22.192068    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal"
Jul 18 16:36:22.196789 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:22.196773    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal"
Jul 18 16:36:22.252407 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:22.252370    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-77.ec2.internal\" not found"
Jul 18 16:36:22.352991 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:22.352962    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-77.ec2.internal\" not found"
Jul 18 16:36:22.453598 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:22.453535    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-77.ec2.internal\" not found"
Jul 18 16:36:22.554236 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:22.554204    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-77.ec2.internal\" not found"
Jul 18 16:36:22.605795 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:22.605765    2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 18 16:36:22.606220 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:22.605928    2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 18 16:36:22.655329 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:22.655304    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-77.ec2.internal\" not found"
Jul 18 16:36:22.667208 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:22.667158    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-17 16:31:22 +0000 UTC" deadline="2028-02-26 04:03:28.550443954 +0000 UTC"
Jul 18 16:36:22.667208 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:22.667200    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14099h27m5.883247228s"
Jul 18 16:36:22.756376 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:22.756319    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-77.ec2.internal\" not found"
Jul 18 16:36:22.809922 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:22.809892    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d6ad110b232b65a608991bf5f07dc7.slice/crio-d23640357071cfb062bb1be788e04c39229f48eafb03b5a3880816c35df7a742 WatchSource:0}: Error finding container d23640357071cfb062bb1be788e04c39229f48eafb03b5a3880816c35df7a742: Status 404 returned error can't find the container with id d23640357071cfb062bb1be788e04c39229f48eafb03b5a3880816c35df7a742
Jul 18 16:36:22.810204 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:22.810182    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00913a0b98636dd43acae671203e933.slice/crio-7e24796fa62874f718f075f1a0f38a44b6d9ea2c014cc10367e63010c98e6457 WatchSource:0}: Error finding container 7e24796fa62874f718f075f1a0f38a44b6d9ea2c014cc10367e63010c98e6457: Status 404 returned error can't find the container with id 7e24796fa62874f718f075f1a0f38a44b6d9ea2c014cc10367e63010c98e6457
Jul 18 16:36:22.815905 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:22.815888    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 16:36:22.839314 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:22.839298    2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 16:36:22.860165 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:22.860142    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal"
Jul 18 16:36:22.873411 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:22.871997    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 18 16:36:22.875609 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:22.875592    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal"
Jul 18 16:36:22.884474 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:22.884450    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 18 16:36:23.124960 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.124887    2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 16:36:23.143018 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.142995    2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 16:36:23.644503 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.644462    2554 apiserver.go:52] "Watching apiserver"
Jul 18 16:36:23.653702 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.653607    2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 18 16:36:23.654180 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.654154    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-kf9nq","openshift-cluster-node-tuning-operator/tuned-wfz6t","openshift-dns/node-resolver-sz6hk","openshift-image-registry/node-ca-6jtwc","openshift-multus/multus-additional-cni-plugins-ss97s","openshift-multus/multus-pvzgv","openshift-network-diagnostics/network-check-target-nx8cd","kube-system/global-pull-secret-syncer-wzt7c","kube-system/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal","openshift-multus/network-metrics-daemon-hczmz","openshift-network-operator/iptables-alerter-k7dtb","openshift-ovn-kubernetes/ovnkube-node-hzpm6"]
Jul 18 16:36:23.657249 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.657224    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:23.657354 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:23.657317    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:23.659993 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.659964    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k7dtb"
Jul 18 16:36:23.660548 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.660522    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 18 16:36:23.662316 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.662288    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sz6hk"
Jul 18 16:36:23.662923 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.662901    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 18 16:36:23.663213 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.663194    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mdp7k\""
Jul 18 16:36:23.663296 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.663209    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 18 16:36:23.663296 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.663236    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 18 16:36:23.664549 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.664530    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 18 16:36:23.664723 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.664705    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-25lps\""
Jul 18 16:36:23.665407 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.665020    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6jtwc"
Jul 18 16:36:23.665407 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.665046    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 18 16:36:23.667280 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.667261    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 18 16:36:23.667507 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.667467    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 18 16:36:23.667611 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.667514    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-th6cn\""
Jul 18 16:36:23.667611 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.667548    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 18 16:36:23.667838 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.667815    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.670117 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.670098    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.671499 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.671232    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 18 16:36:23.671499 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.671250    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 18 16:36:23.671499 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.671275    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 18 16:36:23.671499 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.671388    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q7xvg\""
Jul 18 16:36:23.671765 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.671571    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 18 16:36:23.671946 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.671901    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 18 16:36:23.672237 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.672219    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 18 16:36:23.672572 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.672551    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-j7zrw\""
Jul 18 16:36:23.672940 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.672922    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.673562 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.673344    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlzsm\" (UniqueName: \"kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm\") pod \"network-check-target-nx8cd\" (UID: \"0dfc5886-2c51-4274-a603-260a4d3b4688\") " pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:23.673562 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.673436    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce1c94b6-1296-4d2c-bcff-6797174392c3-iptables-alerter-script\") pod \"iptables-alerter-k7dtb\" (UID: \"ce1c94b6-1296-4d2c-bcff-6797174392c3\") " pod="openshift-network-operator/iptables-alerter-k7dtb"
Jul 18 16:36:23.673562 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.673475    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c71dd87-f690-4c8e-b286-6eff3854fd93-hosts-file\") pod \"node-resolver-sz6hk\" (UID: \"8c71dd87-f690-4c8e-b286-6eff3854fd93\") " pod="openshift-dns/node-resolver-sz6hk"
Jul 18 16:36:23.673562 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.673512    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h594c\" (UniqueName: \"kubernetes.io/projected/8c71dd87-f690-4c8e-b286-6eff3854fd93-kube-api-access-h594c\") pod \"node-resolver-sz6hk\" (UID: \"8c71dd87-f690-4c8e-b286-6eff3854fd93\") " pod="openshift-dns/node-resolver-sz6hk"
Jul 18 16:36:23.673797 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.673759    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce1c94b6-1296-4d2c-bcff-6797174392c3-host-slash\") pod \"iptables-alerter-k7dtb\" (UID: \"ce1c94b6-1296-4d2c-bcff-6797174392c3\") " pod="openshift-network-operator/iptables-alerter-k7dtb"
Jul 18 16:36:23.674164 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.674145    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rnlc\" (UniqueName: \"kubernetes.io/projected/ce1c94b6-1296-4d2c-bcff-6797174392c3-kube-api-access-8rnlc\") pod \"iptables-alerter-k7dtb\" (UID: \"ce1c94b6-1296-4d2c-bcff-6797174392c3\") " pod="openshift-network-operator/iptables-alerter-k7dtb"
Jul 18 16:36:23.674238 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.674190    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c71dd87-f690-4c8e-b286-6eff3854fd93-tmp-dir\") pod \"node-resolver-sz6hk\" (UID: \"8c71dd87-f690-4c8e-b286-6eff3854fd93\") " pod="openshift-dns/node-resolver-sz6hk"
Jul 18 16:36:23.678596 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.678576    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 18 16:36:23.678763 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.678722    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kf9nq"
Jul 18 16:36:23.678835 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.678804    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 18 16:36:23.679060 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.679039    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ds79f\""
Jul 18 16:36:23.680516 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.680093    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 18 16:36:23.680516 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.680209    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 18 16:36:23.680516 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.680247    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 18 16:36:23.680516 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.680276    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 18 16:36:23.680516 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.680411    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 18 16:36:23.681765 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.681712    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 18 16:36:23.681936 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.681914    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.682083 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.681966    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-twmsc\""
Jul 18 16:36:23.683728 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.683707    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 18 16:36:23.684533 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.684175    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 18 16:36:23.685478 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.684786    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 18 16:36:23.685478 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.684788    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z2lp5\""
Jul 18 16:36:23.685478 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.685291    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.687480 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.687455    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 18 16:36:23.688032 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.688012    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 18 16:36:23.688894 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.688273    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n9z45\""
Jul 18 16:36:23.688894 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.688482    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 18 16:36:23.689402 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.689383    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:23.689598 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:23.689566    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:23.690572 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.690551    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:23.690667 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:23.690623    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:23.697871 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.697824    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v4lm2"
Jul 18 16:36:23.706350 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.706330    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v4lm2"
Jul 18 16:36:23.765177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.763267    2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 18 16:36:23.767302 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.767250    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal" event={"ID":"d5d6ad110b232b65a608991bf5f07dc7","Type":"ContainerStarted","Data":"d23640357071cfb062bb1be788e04c39229f48eafb03b5a3880816c35df7a742"}
Jul 18 16:36:23.768567 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.768535    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal" event={"ID":"a00913a0b98636dd43acae671203e933","Type":"ContainerStarted","Data":"7e24796fa62874f718f075f1a0f38a44b6d9ea2c014cc10367e63010c98e6457"}
Jul 18 16:36:23.774420 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774398    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kxd7\" (UniqueName: \"kubernetes.io/projected/7d87a70a-de90-4227-8a88-9a20bc99967e-kube-api-access-5kxd7\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.774538 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774432    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-device-dir\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.774538 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774461    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c71dd87-f690-4c8e-b286-6eff3854fd93-tmp-dir\") pod \"node-resolver-sz6hk\" (UID: \"8c71dd87-f690-4c8e-b286-6eff3854fd93\") " pod="openshift-dns/node-resolver-sz6hk"
Jul 18 16:36:23.774538 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774489    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce1c94b6-1296-4d2c-bcff-6797174392c3-iptables-alerter-script\") pod \"iptables-alerter-k7dtb\" (UID: \"ce1c94b6-1296-4d2c-bcff-6797174392c3\") " pod="openshift-network-operator/iptables-alerter-k7dtb"
Jul 18 16:36:23.774538 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774515    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-lib-modules\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.774753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774537    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/087efd0f-84d1-46e1-b971-a5126ec5b252-cnibin\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.774753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774562    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-var-lib-cni-bin\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.774753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774590    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-var-lib-kubelet\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.774753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774614    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-cni-netd\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.774753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774643    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce1c94b6-1296-4d2c-bcff-6797174392c3-host-slash\") pod \"iptables-alerter-k7dtb\" (UID: \"ce1c94b6-1296-4d2c-bcff-6797174392c3\") " pod="openshift-network-operator/iptables-alerter-k7dtb"
Jul 18 16:36:23.774753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774669    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-systemd-units\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.774753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774693    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-run-systemd\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.774753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774718    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-modprobe-d\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.774753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774742    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a99a42b-8462-4cc8-9512-e9316a46672f-kubelet-config\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:23.775217 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774767    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/087efd0f-84d1-46e1-b971-a5126ec5b252-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.775217 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774793    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/087efd0f-84d1-46e1-b971-a5126ec5b252-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.775217 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774810    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c71dd87-f690-4c8e-b286-6eff3854fd93-tmp-dir\") pod \"node-resolver-sz6hk\" (UID: \"8c71dd87-f690-4c8e-b286-6eff3854fd93\") " pod="openshift-dns/node-resolver-sz6hk"
Jul 18 16:36:23.775217 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774818    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-kubelet\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.775217 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774844    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/087efd0f-84d1-46e1-b971-a5126ec5b252-system-cni-dir\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.775217 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774869    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-multus-socket-dir-parent\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.775217 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.774978    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce1c94b6-1296-4d2c-bcff-6797174392c3-host-slash\") pod \"iptables-alerter-k7dtb\" (UID: \"ce1c94b6-1296-4d2c-bcff-6797174392c3\") " pod="openshift-network-operator/iptables-alerter-k7dtb"
Jul 18 16:36:23.775217 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775083    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-node-log\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.775217 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775119    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d87a70a-de90-4227-8a88-9a20bc99967e-env-overrides\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.775217 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775146    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-socket-dir\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.775217 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775172    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hzbd\" (UniqueName: \"kubernetes.io/projected/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-kube-api-access-8hzbd\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.775217 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775197    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-run-ovn\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.775217 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775220    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-log-socket\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.775841 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775319    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:23.775841 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775424    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-systemd\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.775841 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775482    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z49ft\" (UniqueName: \"kubernetes.io/projected/a48a86ca-3a85-46b8-ada3-0125ec0e9546-kube-api-access-z49ft\") pod \"node-ca-6jtwc\" (UID: \"a48a86ca-3a85-46b8-ada3-0125ec0e9546\") " pod="openshift-image-registry/node-ca-6jtwc"
Jul 18 16:36:23.775841 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775509    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce1c94b6-1296-4d2c-bcff-6797174392c3-iptables-alerter-script\") pod \"iptables-alerter-k7dtb\" (UID: \"ce1c94b6-1296-4d2c-bcff-6797174392c3\") " pod="openshift-network-operator/iptables-alerter-k7dtb"
Jul 18 16:36:23.775841 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775568    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-cni-bin\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.775841 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775612    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-sys-fs\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.775841 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775649    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/087efd0f-84d1-46e1-b971-a5126ec5b252-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.775841 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775677    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-var-lib-cni-multus\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.775841 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775701    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-hostroot\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.775841 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775723    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-multus-conf-dir\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.775841 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775750    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-etc-kubernetes\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.775841 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775773    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-run-netns\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775873    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-var-lib-openvswitch\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775917    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjrrn\" (UniqueName: \"kubernetes.io/projected/01b34321-bf74-4a38-9791-c85a14974eff-kube-api-access-hjrrn\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775943    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f186e77b-ccfd-461c-8cb8-e4f3a746fbeb-agent-certs\") pod \"konnectivity-agent-kf9nq\" (UID: \"f186e77b-ccfd-461c-8cb8-e4f3a746fbeb\") " pod="kube-system/konnectivity-agent-kf9nq"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775970    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-sysctl-conf\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.775994    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-host\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776019    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/087efd0f-84d1-46e1-b971-a5126ec5b252-os-release\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776036    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-run-multus-certs\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776066    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-registration-dir\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776087    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-tmp\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776109    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a99a42b-8462-4cc8-9512-e9316a46672f-dbus\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776133    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/544a8f18-3d52-422e-8a65-b5b488e82bdd-cni-binary-copy\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776158    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhzb\" (UniqueName: \"kubernetes.io/projected/2f9ff934-e093-4adb-b818-279e624555de-kube-api-access-dlhzb\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776186    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c71dd87-f690-4c8e-b286-6eff3854fd93-hosts-file\") pod \"node-resolver-sz6hk\" (UID: \"8c71dd87-f690-4c8e-b286-6eff3854fd93\") " pod="openshift-dns/node-resolver-sz6hk"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776210    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-sysctl-d\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776317    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c71dd87-f690-4c8e-b286-6eff3854fd93-hosts-file\") pod \"node-resolver-sz6hk\" (UID: \"8c71dd87-f690-4c8e-b286-6eff3854fd93\") " pod="openshift-dns/node-resolver-sz6hk"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776347    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-os-release\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.776448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776378    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzsm\" (UniqueName: \"kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm\") pod \"network-check-target-nx8cd\" (UID: \"0dfc5886-2c51-4274-a603-260a4d3b4688\") " pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776407    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-sys\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776436    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-var-lib-kubelet\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776458    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-run-ovn-kubernetes\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776486    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-etc-selinux\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776512    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h594c\" (UniqueName: \"kubernetes.io/projected/8c71dd87-f690-4c8e-b286-6eff3854fd93-kube-api-access-h594c\") pod \"node-resolver-sz6hk\" (UID: \"8c71dd87-f690-4c8e-b286-6eff3854fd93\") " pod="openshift-dns/node-resolver-sz6hk"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776536    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776558    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a48a86ca-3a85-46b8-ada3-0125ec0e9546-host\") pod \"node-ca-6jtwc\" (UID: \"a48a86ca-3a85-46b8-ada3-0125ec0e9546\") " pod="openshift-image-registry/node-ca-6jtwc"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776581    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvzvd\" (UniqueName: \"kubernetes.io/projected/544a8f18-3d52-422e-8a65-b5b488e82bdd-kube-api-access-qvzvd\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776708    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-etc-openvswitch\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776752    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776777    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-kubernetes\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776800    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-multus-cni-dir\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776820    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d87a70a-de90-4227-8a88-9a20bc99967e-ovn-node-metrics-cert\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776846    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rnlc\" (UniqueName: \"kubernetes.io/projected/ce1c94b6-1296-4d2c-bcff-6797174392c3-kube-api-access-8rnlc\") pod \"iptables-alerter-k7dtb\" (UID: \"ce1c94b6-1296-4d2c-bcff-6797174392c3\") " pod="openshift-network-operator/iptables-alerter-k7dtb"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776893    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f186e77b-ccfd-461c-8cb8-e4f3a746fbeb-konnectivity-ca\") pod \"konnectivity-agent-kf9nq\" (UID: \"f186e77b-ccfd-461c-8cb8-e4f3a746fbeb\") " pod="kube-system/konnectivity-agent-kf9nq"
Jul 18 16:36:23.777248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776911    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-tuned\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.777807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776926    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a48a86ca-3a85-46b8-ada3-0125ec0e9546-serviceca\") pod \"node-ca-6jtwc\" (UID: \"a48a86ca-3a85-46b8-ada3-0125ec0e9546\") " pod="openshift-image-registry/node-ca-6jtwc"
Jul 18 16:36:23.777807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776943    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/087efd0f-84d1-46e1-b971-a5126ec5b252-cni-binary-copy\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.777807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776975    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8g4r\" (UniqueName: \"kubernetes.io/projected/087efd0f-84d1-46e1-b971-a5126ec5b252-kube-api-access-l8g4r\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.777807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.776996    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-cnibin\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.777807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.777020    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-run-netns\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.777807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.777046    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-sysconfig\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.777807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.777087    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-slash\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.777807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.777153    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-run-openvswitch\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.777807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.777173    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d87a70a-de90-4227-8a88-9a20bc99967e-ovnkube-config\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.777807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.777188    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d87a70a-de90-4227-8a88-9a20bc99967e-ovnkube-script-lib\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.777807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.777247    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-run\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.777807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.777268    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-system-cni-dir\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.777807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.777282    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-run-k8s-cni-cncf-io\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.777807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.777303    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/544a8f18-3d52-422e-8a65-b5b488e82bdd-multus-daemon-config\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.777807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.777344    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.782410 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:23.782391    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 16:36:23.782533 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:23.782414    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 16:36:23.782533 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:23.782427    2554 projected.go:196] Error preparing data for projected volume kube-api-access-zlzsm for pod openshift-network-diagnostics/network-check-target-nx8cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 16:36:23.782533 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:23.782501    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm podName:0dfc5886-2c51-4274-a603-260a4d3b4688 nodeName:}" failed. No retries permitted until 2026-07-18 16:36:24.28247557 +0000 UTC m=+2.980306246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zlzsm" (UniqueName: "kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm") pod "network-check-target-nx8cd" (UID: "0dfc5886-2c51-4274-a603-260a4d3b4688") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 16:36:23.782712 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.782615    2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 18 16:36:23.786294 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.786275    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rnlc\" (UniqueName: \"kubernetes.io/projected/ce1c94b6-1296-4d2c-bcff-6797174392c3-kube-api-access-8rnlc\") pod \"iptables-alerter-k7dtb\" (UID: \"ce1c94b6-1296-4d2c-bcff-6797174392c3\") " pod="openshift-network-operator/iptables-alerter-k7dtb"
Jul 18 16:36:23.786294 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.786289    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h594c\" (UniqueName: \"kubernetes.io/projected/8c71dd87-f690-4c8e-b286-6eff3854fd93-kube-api-access-h594c\") pod \"node-resolver-sz6hk\" (UID: \"8c71dd87-f690-4c8e-b286-6eff3854fd93\") " pod="openshift-dns/node-resolver-sz6hk"
Jul 18 16:36:23.878121 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878079    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-os-release\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.878308 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878162    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-sys\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.878308 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878195    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-var-lib-kubelet\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.878308 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878226    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-run-ovn-kubernetes\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.878308 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878252    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-etc-selinux\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.878308 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878276    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:23.878308 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878282    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-var-lib-kubelet\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.878308 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878299    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a48a86ca-3a85-46b8-ada3-0125ec0e9546-host\") pod \"node-ca-6jtwc\" (UID: \"a48a86ca-3a85-46b8-ada3-0125ec0e9546\") " pod="openshift-image-registry/node-ca-6jtwc"
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878274    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-sys\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878326    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvzvd\" (UniqueName: \"kubernetes.io/projected/544a8f18-3d52-422e-8a65-b5b488e82bdd-kube-api-access-qvzvd\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878343    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-run-ovn-kubernetes\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878365    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-etc-openvswitch\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878394    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:23.878401    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878406    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a48a86ca-3a85-46b8-ada3-0125ec0e9546-host\") pod \"node-ca-6jtwc\" (UID: \"a48a86ca-3a85-46b8-ada3-0125ec0e9546\") " pod="openshift-image-registry/node-ca-6jtwc"
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878196    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-os-release\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878421    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-kubernetes\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878425    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-etc-selinux\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878445    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-multus-cni-dir\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878460    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-etc-openvswitch\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:23.878465    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret podName:5a99a42b-8462-4cc8-9512-e9316a46672f nodeName:}" failed. No retries permitted until 2026-07-18 16:36:24.378444308 +0000 UTC m=+3.076274988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret") pod "global-pull-secret-syncer-wzt7c" (UID: "5a99a42b-8462-4cc8-9512-e9316a46672f") : object "kube-system"/"original-pull-secret" not registered
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878495    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878497    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-kubernetes\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878505    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d87a70a-de90-4227-8a88-9a20bc99967e-ovn-node-metrics-cert\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.878591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878545    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f186e77b-ccfd-461c-8cb8-e4f3a746fbeb-konnectivity-ca\") pod \"konnectivity-agent-kf9nq\" (UID: \"f186e77b-ccfd-461c-8cb8-e4f3a746fbeb\") " pod="kube-system/konnectivity-agent-kf9nq"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878562    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-tuned\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878577    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a48a86ca-3a85-46b8-ada3-0125ec0e9546-serviceca\") pod \"node-ca-6jtwc\" (UID: \"a48a86ca-3a85-46b8-ada3-0125ec0e9546\") " pod="openshift-image-registry/node-ca-6jtwc"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878594    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/087efd0f-84d1-46e1-b971-a5126ec5b252-cni-binary-copy\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878612    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8g4r\" (UniqueName: \"kubernetes.io/projected/087efd0f-84d1-46e1-b971-a5126ec5b252-kube-api-access-l8g4r\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878637    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-multus-cni-dir\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878642    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-cnibin\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878674    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-cnibin\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878681    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-run-netns\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878706    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-sysconfig\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878730    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-slash\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878756    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-run-openvswitch\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878782    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d87a70a-de90-4227-8a88-9a20bc99967e-ovnkube-config\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878806    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d87a70a-de90-4227-8a88-9a20bc99967e-ovnkube-script-lib\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878832    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-run\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878857    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-system-cni-dir\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878881    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-run-k8s-cni-cncf-io\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878903    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/544a8f18-3d52-422e-8a65-b5b488e82bdd-multus-daemon-config\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.879419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878948    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.878979    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kxd7\" (UniqueName: \"kubernetes.io/projected/7d87a70a-de90-4227-8a88-9a20bc99967e-kube-api-access-5kxd7\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879095    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a48a86ca-3a85-46b8-ada3-0125ec0e9546-serviceca\") pod \"node-ca-6jtwc\" (UID: \"a48a86ca-3a85-46b8-ada3-0125ec0e9546\") " pod="openshift-image-registry/node-ca-6jtwc"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879114    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-device-dir\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879148    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-lib-modules\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879175    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/087efd0f-84d1-46e1-b971-a5126ec5b252-cnibin\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879203    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-var-lib-cni-bin\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879230    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-var-lib-kubelet\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879245    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-system-cni-dir\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879255    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-cni-netd\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879282    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-systemd-units\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879291    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-run-k8s-cni-cncf-io\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879307    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-run-systemd\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879332    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-modprobe-d\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879357    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a99a42b-8462-4cc8-9512-e9316a46672f-kubelet-config\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879383    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/087efd0f-84d1-46e1-b971-a5126ec5b252-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879409    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/087efd0f-84d1-46e1-b971-a5126ec5b252-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.880358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879433    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-kubelet\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879436    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/087efd0f-84d1-46e1-b971-a5126ec5b252-cni-binary-copy\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879459    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/087efd0f-84d1-46e1-b971-a5126ec5b252-system-cni-dir\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879489    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-multus-socket-dir-parent\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879515    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-node-log\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879520    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-device-dir\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879537    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d87a70a-de90-4227-8a88-9a20bc99967e-env-overrides\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879563    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-socket-dir\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879593    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hzbd\" (UniqueName: \"kubernetes.io/projected/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-kube-api-access-8hzbd\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879621    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-run-ovn\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879646    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-log-socket\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879649    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-lib-modules\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879671    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879698    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/087efd0f-84d1-46e1-b971-a5126ec5b252-cnibin\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879699    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-systemd\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879746    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-var-lib-cni-bin\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879753    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z49ft\" (UniqueName: \"kubernetes.io/projected/a48a86ca-3a85-46b8-ada3-0125ec0e9546-kube-api-access-z49ft\") pod \"node-ca-6jtwc\" (UID: \"a48a86ca-3a85-46b8-ada3-0125ec0e9546\") " pod="openshift-image-registry/node-ca-6jtwc"
Jul 18 16:36:23.885177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879203    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-run\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879783    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-cni-bin\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879801    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-var-lib-kubelet\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879807    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-sys-fs\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879847    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/087efd0f-84d1-46e1-b971-a5126ec5b252-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879859    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-sys-fs\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879899    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/544a8f18-3d52-422e-8a65-b5b488e82bdd-multus-daemon-config\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879937    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-systemd-units\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879941    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-var-lib-cni-multus\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879940    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d87a70a-de90-4227-8a88-9a20bc99967e-ovnkube-script-lib\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879971    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-hostroot\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879973    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879980    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-run-systemd\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880001    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-multus-conf-dir\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880016    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-run-netns\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880028    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-etc-kubernetes\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880058    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-run-netns\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880071    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-slash\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.886125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880098    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-modprobe-d\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880103    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-run-netns\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880097    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-run-openvswitch\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.879941    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f186e77b-ccfd-461c-8cb8-e4f3a746fbeb-konnectivity-ca\") pod \"konnectivity-agent-kf9nq\" (UID: \"f186e77b-ccfd-461c-8cb8-e4f3a746fbeb\") " pod="kube-system/konnectivity-agent-kf9nq"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880120    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-var-lib-openvswitch\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880153    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-var-lib-cni-multus\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880155    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-hostroot\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880193    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-var-lib-openvswitch\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880215    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-multus-conf-dir\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880235    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-etc-kubernetes\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880265    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-sysconfig\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880326    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-systemd\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880336    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-socket-dir\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880153    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjrrn\" (UniqueName: \"kubernetes.io/projected/01b34321-bf74-4a38-9791-c85a14974eff-kube-api-access-hjrrn\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880385    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-log-socket\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880387    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f186e77b-ccfd-461c-8cb8-e4f3a746fbeb-agent-certs\") pod \"konnectivity-agent-kf9nq\" (UID: \"f186e77b-ccfd-461c-8cb8-e4f3a746fbeb\") " pod="kube-system/konnectivity-agent-kf9nq"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880406    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a99a42b-8462-4cc8-9512-e9316a46672f-kubelet-config\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:23.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880412    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/087efd0f-84d1-46e1-b971-a5126ec5b252-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880418    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-sysctl-conf\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:23.880474    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880476    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-host\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880504    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/087efd0f-84d1-46e1-b971-a5126ec5b252-os-release\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:23.880522    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs podName:2f9ff934-e093-4adb-b818-279e624555de nodeName:}" failed. No retries permitted until 2026-07-18 16:36:24.380504973 +0000 UTC m=+3.078335640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs") pod "network-metrics-daemon-hczmz" (UID: "2f9ff934-e093-4adb-b818-279e624555de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880552    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-run-multus-certs\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880564    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/087efd0f-84d1-46e1-b971-a5126ec5b252-os-release\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880565    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d87a70a-de90-4227-8a88-9a20bc99967e-ovnkube-config\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880582    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-registration-dir\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880614    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-tmp\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880630    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-host\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880638    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-multus-socket-dir-parent\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880641    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a99a42b-8462-4cc8-9512-e9316a46672f-dbus\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880692    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a99a42b-8462-4cc8-9512-e9316a46672f-dbus\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880734    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-kubelet\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880778    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/087efd0f-84d1-46e1-b971-a5126ec5b252-system-cni-dir\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.888080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880819    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-cni-netd\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880891    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/087efd0f-84d1-46e1-b971-a5126ec5b252-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880984    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/087efd0f-84d1-46e1-b971-a5126ec5b252-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.881055    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01b34321-bf74-4a38-9791-c85a14974eff-registration-dir\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.881099    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/544a8f18-3d52-422e-8a65-b5b488e82bdd-host-run-multus-certs\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.881228    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/544a8f18-3d52-422e-8a65-b5b488e82bdd-cni-binary-copy\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.881247    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-sysctl-conf\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.880692    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/544a8f18-3d52-422e-8a65-b5b488e82bdd-cni-binary-copy\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.881295    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-host-cni-bin\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.881303    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-node-log\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.881306    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhzb\" (UniqueName: \"kubernetes.io/projected/2f9ff934-e093-4adb-b818-279e624555de-kube-api-access-dlhzb\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.881340    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-sysctl-d\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.881345    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d87a70a-de90-4227-8a88-9a20bc99967e-run-ovn\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.881254    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d87a70a-de90-4227-8a88-9a20bc99967e-env-overrides\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.881479    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-sysctl-d\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.881542    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-etc-tuned\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.881763    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d87a70a-de90-4227-8a88-9a20bc99967e-ovn-node-metrics-cert\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.883447    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f186e77b-ccfd-461c-8cb8-e4f3a746fbeb-agent-certs\") pod \"konnectivity-agent-kf9nq\" (UID: \"f186e77b-ccfd-461c-8cb8-e4f3a746fbeb\") " pod="kube-system/konnectivity-agent-kf9nq"
Jul 18 16:36:23.889011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.883573    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-tmp\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.891053 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.890997    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvzvd\" (UniqueName: \"kubernetes.io/projected/544a8f18-3d52-422e-8a65-b5b488e82bdd-kube-api-access-qvzvd\") pod \"multus-pvzgv\" (UID: \"544a8f18-3d52-422e-8a65-b5b488e82bdd\") " pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:23.892103 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.892056    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjrrn\" (UniqueName: \"kubernetes.io/projected/01b34321-bf74-4a38-9791-c85a14974eff-kube-api-access-hjrrn\") pod \"aws-ebs-csi-driver-node-nl4p5\" (UID: \"01b34321-bf74-4a38-9791-c85a14974eff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:23.893175 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.893098    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kxd7\" (UniqueName: \"kubernetes.io/projected/7d87a70a-de90-4227-8a88-9a20bc99967e-kube-api-access-5kxd7\") pod \"ovnkube-node-hzpm6\" (UID: \"7d87a70a-de90-4227-8a88-9a20bc99967e\") " pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:23.893599 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.893574    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhzb\" (UniqueName: \"kubernetes.io/projected/2f9ff934-e093-4adb-b818-279e624555de-kube-api-access-dlhzb\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:23.893921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.893893    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hzbd\" (UniqueName: \"kubernetes.io/projected/ac112e8c-d18e-4f1b-8d1e-8675a88cac3f-kube-api-access-8hzbd\") pod \"tuned-wfz6t\" (UID: \"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f\") " pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:23.894177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.894151    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8g4r\" (UniqueName: \"kubernetes.io/projected/087efd0f-84d1-46e1-b971-a5126ec5b252-kube-api-access-l8g4r\") pod \"multus-additional-cni-plugins-ss97s\" (UID: \"087efd0f-84d1-46e1-b971-a5126ec5b252\") " pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:23.895779 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.895757    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z49ft\" (UniqueName: \"kubernetes.io/projected/a48a86ca-3a85-46b8-ada3-0125ec0e9546-kube-api-access-z49ft\") pod \"node-ca-6jtwc\" (UID: \"a48a86ca-3a85-46b8-ada3-0125ec0e9546\") " pod="openshift-image-registry/node-ca-6jtwc"
Jul 18 16:36:23.978997 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.978920    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k7dtb"
Jul 18 16:36:23.986850 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.986825    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sz6hk"
Jul 18 16:36:23.995629 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:23.995605    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6jtwc"
Jul 18 16:36:24.004088 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.004066    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ss97s"
Jul 18 16:36:24.011193 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.011174    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pvzgv"
Jul 18 16:36:24.019013 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.018993    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:24.028020 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.027867    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kf9nq"
Jul 18 16:36:24.038013 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.037980    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wfz6t"
Jul 18 16:36:24.039476 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.039459    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 18 16:36:24.042310 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.042286    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5"
Jul 18 16:36:24.284914 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.284812    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzsm\" (UniqueName: \"kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm\") pod \"network-check-target-nx8cd\" (UID: \"0dfc5886-2c51-4274-a603-260a4d3b4688\") " pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:24.285085 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:24.285005    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 16:36:24.285085 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:24.285026    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 16:36:24.285085 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:24.285037    2554 projected.go:196] Error preparing data for projected volume kube-api-access-zlzsm for pod openshift-network-diagnostics/network-check-target-nx8cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 16:36:24.285261 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:24.285099    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm podName:0dfc5886-2c51-4274-a603-260a4d3b4688 nodeName:}" failed. No retries permitted until 2026-07-18 16:36:25.285081831 +0000 UTC m=+3.982912515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zlzsm" (UniqueName: "kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm") pod "network-check-target-nx8cd" (UID: "0dfc5886-2c51-4274-a603-260a4d3b4688") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 16:36:24.385947 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.385908    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:24.386152 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.386001    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:24.386152 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:24.386072    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 16:36:24.386152 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:24.386104    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 16:36:24.386152 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:24.386144    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs podName:2f9ff934-e093-4adb-b818-279e624555de nodeName:}" failed. No retries permitted until 2026-07-18 16:36:25.386123482 +0000 UTC m=+4.083954165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs") pod "network-metrics-daemon-hczmz" (UID: "2f9ff934-e093-4adb-b818-279e624555de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 16:36:24.386328 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:24.386161    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret podName:5a99a42b-8462-4cc8-9512-e9316a46672f nodeName:}" failed. No retries permitted until 2026-07-18 16:36:25.386153948 +0000 UTC m=+4.083984612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret") pod "global-pull-secret-syncer-wzt7c" (UID: "5a99a42b-8462-4cc8-9512-e9316a46672f") : object "kube-system"/"original-pull-secret" not registered
Jul 18 16:36:24.711185 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.711097    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 16:31:24 +0000 UTC" deadline="2028-01-17 12:58:02.04379987 +0000 UTC"
Jul 18 16:36:24.711185 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.711143    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13148h21m37.332659714s"
Jul 18 16:36:24.734476 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:24.734433    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac112e8c_d18e_4f1b_8d1e_8675a88cac3f.slice/crio-ae084fff64157b655f9ad64597eae64e698480ee426583c01f2ac97c915864be WatchSource:0}: Error finding container ae084fff64157b655f9ad64597eae64e698480ee426583c01f2ac97c915864be: Status 404 returned error can't find the container with id ae084fff64157b655f9ad64597eae64e698480ee426583c01f2ac97c915864be
Jul 18 16:36:24.736843 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:24.736815    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48a86ca_3a85_46b8_ada3_0125ec0e9546.slice/crio-c780a885f9f12b51f1bfd06d6893fba252aaf0e273326ef5e59f9f1b97fd98b3 WatchSource:0}: Error finding container c780a885f9f12b51f1bfd06d6893fba252aaf0e273326ef5e59f9f1b97fd98b3: Status 404 returned error can't find the container with id c780a885f9f12b51f1bfd06d6893fba252aaf0e273326ef5e59f9f1b97fd98b3
Jul 18 16:36:24.739559 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:24.739530    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf186e77b_ccfd_461c_8cb8_e4f3a746fbeb.slice/crio-1da6c0e546320c7de863e38bbe880c745f358a310c0289897191d280ca1379fa WatchSource:0}: Error finding container 1da6c0e546320c7de863e38bbe880c745f358a310c0289897191d280ca1379fa: Status 404 returned error can't find the container with id 1da6c0e546320c7de863e38bbe880c745f358a310c0289897191d280ca1379fa
Jul 18 16:36:24.740409 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:24.740385    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c71dd87_f690_4c8e_b286_6eff3854fd93.slice/crio-bfc9649a4058a285d7e1795ff29154c554b7d3e9e228804b18b58a63b2975f1f WatchSource:0}: Error finding container bfc9649a4058a285d7e1795ff29154c554b7d3e9e228804b18b58a63b2975f1f: Status 404 returned error can't find the container with id bfc9649a4058a285d7e1795ff29154c554b7d3e9e228804b18b58a63b2975f1f
Jul 18 16:36:24.741286 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:24.741213    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087efd0f_84d1_46e1_b971_a5126ec5b252.slice/crio-5deeaf34be42fff433b323eeaa7b8990924262e1fb3bcb3bcb7252bd18facc76 WatchSource:0}: Error finding container 5deeaf34be42fff433b323eeaa7b8990924262e1fb3bcb3bcb7252bd18facc76: Status 404 returned error can't find the container with id 5deeaf34be42fff433b323eeaa7b8990924262e1fb3bcb3bcb7252bd18facc76
Jul 18 16:36:24.742094 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:24.742071    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1c94b6_1296_4d2c_bcff_6797174392c3.slice/crio-782d920498e448c1cb57905fade16381b3191b8f8372ddffc3a28eb898422e7b WatchSource:0}: Error finding container 782d920498e448c1cb57905fade16381b3191b8f8372ddffc3a28eb898422e7b: Status 404 returned error can't find the container with id 782d920498e448c1cb57905fade16381b3191b8f8372ddffc3a28eb898422e7b
Jul 18 16:36:24.744213 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:24.744119    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544a8f18_3d52_422e_8a65_b5b488e82bdd.slice/crio-c6fe01613072db75d91844d56aa26c5477194cfe930b7a0b07cfdff351473100 WatchSource:0}: Error finding container c6fe01613072db75d91844d56aa26c5477194cfe930b7a0b07cfdff351473100: Status 404 returned error can't find the container with id c6fe01613072db75d91844d56aa26c5477194cfe930b7a0b07cfdff351473100
Jul 18 16:36:24.745157 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:24.745131    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b34321_bf74_4a38_9791_c85a14974eff.slice/crio-88188a6d58ca63e6a689909baf2cc326cbd192e271be0497f2d863e03b41dd54 WatchSource:0}: Error finding container 88188a6d58ca63e6a689909baf2cc326cbd192e271be0497f2d863e03b41dd54: Status 404 returned error can't find the container with id 88188a6d58ca63e6a689909baf2cc326cbd192e271be0497f2d863e03b41dd54
Jul 18 16:36:24.746045 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:24.745992    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d87a70a_de90_4227_8a88_9a20bc99967e.slice/crio-e2e85aeeab1b8c272d494743f023b6ba678a1797b78f7ca1fd4b70f796cdc4d1 WatchSource:0}: Error finding container e2e85aeeab1b8c272d494743f023b6ba678a1797b78f7ca1fd4b70f796cdc4d1: Status 404 returned error can't find the container with id e2e85aeeab1b8c272d494743f023b6ba678a1797b78f7ca1fd4b70f796cdc4d1
Jul 18 16:36:24.771722 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.771689    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pvzgv" event={"ID":"544a8f18-3d52-422e-8a65-b5b488e82bdd","Type":"ContainerStarted","Data":"c6fe01613072db75d91844d56aa26c5477194cfe930b7a0b07cfdff351473100"}
Jul 18 16:36:24.773267 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.773236    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ss97s" event={"ID":"087efd0f-84d1-46e1-b971-a5126ec5b252","Type":"ContainerStarted","Data":"5deeaf34be42fff433b323eeaa7b8990924262e1fb3bcb3bcb7252bd18facc76"}
Jul 18 16:36:24.775644 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.775612    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kf9nq" event={"ID":"f186e77b-ccfd-461c-8cb8-e4f3a746fbeb","Type":"ContainerStarted","Data":"1da6c0e546320c7de863e38bbe880c745f358a310c0289897191d280ca1379fa"}
Jul 18 16:36:24.779914 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.779888    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6" event={"ID":"7d87a70a-de90-4227-8a88-9a20bc99967e","Type":"ContainerStarted","Data":"e2e85aeeab1b8c272d494743f023b6ba678a1797b78f7ca1fd4b70f796cdc4d1"}
Jul 18 16:36:24.782823 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.782797    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5" event={"ID":"01b34321-bf74-4a38-9791-c85a14974eff","Type":"ContainerStarted","Data":"88188a6d58ca63e6a689909baf2cc326cbd192e271be0497f2d863e03b41dd54"}
Jul 18 16:36:24.785321 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.785295    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k7dtb" event={"ID":"ce1c94b6-1296-4d2c-bcff-6797174392c3","Type":"ContainerStarted","Data":"782d920498e448c1cb57905fade16381b3191b8f8372ddffc3a28eb898422e7b"}
Jul 18 16:36:24.786762 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.786735    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6jtwc" event={"ID":"a48a86ca-3a85-46b8-ada3-0125ec0e9546","Type":"ContainerStarted","Data":"c780a885f9f12b51f1bfd06d6893fba252aaf0e273326ef5e59f9f1b97fd98b3"}
Jul 18 16:36:24.787785 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.787762    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wfz6t" event={"ID":"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f","Type":"ContainerStarted","Data":"ae084fff64157b655f9ad64597eae64e698480ee426583c01f2ac97c915864be"}
Jul 18 16:36:24.788764 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:24.788737    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sz6hk" event={"ID":"8c71dd87-f690-4c8e-b286-6eff3854fd93","Type":"ContainerStarted","Data":"bfc9649a4058a285d7e1795ff29154c554b7d3e9e228804b18b58a63b2975f1f"}
Jul 18 16:36:25.294885 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:25.294616    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzsm\" (UniqueName: \"kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm\") pod \"network-check-target-nx8cd\" (UID: \"0dfc5886-2c51-4274-a603-260a4d3b4688\") " pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:25.295103 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:25.294913    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 16:36:25.295103 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:25.294962    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 16:36:25.295103 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:25.294976    2554 projected.go:196] Error preparing data for projected volume kube-api-access-zlzsm for pod openshift-network-diagnostics/network-check-target-nx8cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 16:36:25.295103 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:25.295048    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm podName:0dfc5886-2c51-4274-a603-260a4d3b4688 nodeName:}" failed. No retries permitted until 2026-07-18 16:36:27.295026035 +0000 UTC m=+5.992856714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zlzsm" (UniqueName: "kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm") pod "network-check-target-nx8cd" (UID: "0dfc5886-2c51-4274-a603-260a4d3b4688") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 16:36:25.396835 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:25.395997    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:25.396835 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:25.396094    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:25.396835 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:25.396253    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 16:36:25.396835 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:25.396322    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs podName:2f9ff934-e093-4adb-b818-279e624555de nodeName:}" failed. No retries permitted until 2026-07-18 16:36:27.396299816 +0000 UTC m=+6.094130482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs") pod "network-metrics-daemon-hczmz" (UID: "2f9ff934-e093-4adb-b818-279e624555de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 16:36:25.396835 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:25.396389    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 16:36:25.396835 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:25.396423    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret podName:5a99a42b-8462-4cc8-9512-e9316a46672f nodeName:}" failed. No retries permitted until 2026-07-18 16:36:27.396412154 +0000 UTC m=+6.094242822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret") pod "global-pull-secret-syncer-wzt7c" (UID: "5a99a42b-8462-4cc8-9512-e9316a46672f") : object "kube-system"/"original-pull-secret" not registered
Jul 18 16:36:25.712318 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:25.712220    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 16:31:24 +0000 UTC" deadline="2028-03-16 12:43:03.687657072 +0000 UTC"
Jul 18 16:36:25.712318 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:25.712268    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14564h6m37.975395855s"
Jul 18 16:36:25.763519 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:25.762621    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:25.763519 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:25.762778    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:25.763519 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:25.763228    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:25.763519 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:25.763337    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:25.763519 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:25.763396    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:25.763519 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:25.763468    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:25.804840 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:25.804068    2554 generic.go:356] "Generic (PLEG): container finished" podID="d5d6ad110b232b65a608991bf5f07dc7" containerID="e60f29f1e34b7d9c2bc541663faee7b53f61835903e90b272f624b4b4ba9b1d5" exitCode=0
Jul 18 16:36:25.804840 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:25.804414    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal" event={"ID":"d5d6ad110b232b65a608991bf5f07dc7","Type":"ContainerDied","Data":"e60f29f1e34b7d9c2bc541663faee7b53f61835903e90b272f624b4b4ba9b1d5"}
Jul 18 16:36:25.813855 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:25.813379    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal" event={"ID":"a00913a0b98636dd43acae671203e933","Type":"ContainerStarted","Data":"e298c3f040827274ea9392ed6f1134a2e19935887903ffad403754f98e53eb9e"}
Jul 18 16:36:25.840944 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:25.837857    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-77.ec2.internal" podStartSLOduration=2.837837185 podStartE2EDuration="2.837837185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 16:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:36:25.83240794 +0000 UTC m=+4.530238628" watchObservedRunningTime="2026-07-18 16:36:25.837837185 +0000 UTC m=+4.535667873"
Jul 18 16:36:26.853542 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:26.853094    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal" event={"ID":"d5d6ad110b232b65a608991bf5f07dc7","Type":"ContainerStarted","Data":"e87bdbe33784beded3cf7251bd706206cab01a9dfa7f95922717b69c320cfedc"}
Jul 18 16:36:27.318963 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:27.318213    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzsm\" (UniqueName: \"kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm\") pod \"network-check-target-nx8cd\" (UID: \"0dfc5886-2c51-4274-a603-260a4d3b4688\") " pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:27.318963 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:27.318395    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 16:36:27.318963 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:27.318420    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 16:36:27.318963 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:27.318433    2554 projected.go:196] Error preparing data for projected volume kube-api-access-zlzsm for pod openshift-network-diagnostics/network-check-target-nx8cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 16:36:27.318963 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:27.318506    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm podName:0dfc5886-2c51-4274-a603-260a4d3b4688 nodeName:}" failed. No retries permitted until 2026-07-18 16:36:31.318485409 +0000 UTC m=+10.016316076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zlzsm" (UniqueName: "kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm") pod "network-check-target-nx8cd" (UID: "0dfc5886-2c51-4274-a603-260a4d3b4688") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 16:36:27.419559 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:27.419515    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:27.419727 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:27.419608    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:27.419779 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:27.419763    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 16:36:27.419846 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:27.419829    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret podName:5a99a42b-8462-4cc8-9512-e9316a46672f nodeName:}" failed. No retries permitted until 2026-07-18 16:36:31.419810723 +0000 UTC m=+10.117641393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret") pod "global-pull-secret-syncer-wzt7c" (UID: "5a99a42b-8462-4cc8-9512-e9316a46672f") : object "kube-system"/"original-pull-secret" not registered
Jul 18 16:36:27.419945 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:27.419927    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 16:36:27.420032 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:27.419979    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs podName:2f9ff934-e093-4adb-b818-279e624555de nodeName:}" failed. No retries permitted until 2026-07-18 16:36:31.419966692 +0000 UTC m=+10.117797354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs") pod "network-metrics-daemon-hczmz" (UID: "2f9ff934-e093-4adb-b818-279e624555de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 16:36:27.761068 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:27.760986    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:27.761229 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:27.761140    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:27.761229 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:27.760994    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:27.761353 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:27.761243    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:27.761353 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:27.761305    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:27.761451 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:27.761384    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:29.763606 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:29.762755    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:29.763606 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:29.762904    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:29.763606 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:29.762985    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:29.763606 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:29.763078    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:29.763606 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:29.763123    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:29.763606 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:29.763189    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:31.356304 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:31.356262    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzsm\" (UniqueName: \"kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm\") pod \"network-check-target-nx8cd\" (UID: \"0dfc5886-2c51-4274-a603-260a4d3b4688\") " pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:31.356811 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:31.356464    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 16:36:31.356811 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:31.356493    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 16:36:31.356811 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:31.356509    2554 projected.go:196] Error preparing data for projected volume kube-api-access-zlzsm for pod openshift-network-diagnostics/network-check-target-nx8cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 16:36:31.356811 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:31.356576    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm podName:0dfc5886-2c51-4274-a603-260a4d3b4688 nodeName:}" failed. No retries permitted until 2026-07-18 16:36:39.356558936 +0000 UTC m=+18.054389617 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zlzsm" (UniqueName: "kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm") pod "network-check-target-nx8cd" (UID: "0dfc5886-2c51-4274-a603-260a4d3b4688") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 16:36:31.457088 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:31.456623    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:31.457088 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:31.456704    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:31.457088 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:31.456855    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 16:36:31.457088 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:31.456903    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 16:36:31.457088 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:31.456948    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret podName:5a99a42b-8462-4cc8-9512-e9316a46672f nodeName:}" failed. No retries permitted until 2026-07-18 16:36:39.456928497 +0000 UTC m=+18.154759161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret") pod "global-pull-secret-syncer-wzt7c" (UID: "5a99a42b-8462-4cc8-9512-e9316a46672f") : object "kube-system"/"original-pull-secret" not registered
Jul 18 16:36:31.457088 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:31.456970    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs podName:2f9ff934-e093-4adb-b818-279e624555de nodeName:}" failed. No retries permitted until 2026-07-18 16:36:39.45695892 +0000 UTC m=+18.154789592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs") pod "network-metrics-daemon-hczmz" (UID: "2f9ff934-e093-4adb-b818-279e624555de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 16:36:31.763876 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:31.763776    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:31.764037 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:31.763876    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:31.764037 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:31.763899    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:31.764037 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:31.763989    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:31.764226 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:31.764051    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:31.764226 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:31.764135    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:33.762002 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:33.761975    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:33.762422 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:33.762102    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:33.762474 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:33.762432    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:33.762561 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:33.762521    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:33.762679 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:33.762579    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:33.762771 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:33.762662    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:35.759840 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:35.759807    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:35.760352 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:35.759817    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:35.760352 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:35.759938    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:35.760352 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:35.760030    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:35.760352 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:35.760081    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:35.760352 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:35.760160    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:37.760067 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:37.760034    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:37.760514 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:37.760034    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:37.760514 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:37.760160    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:37.760514 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:37.760034    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:37.760514 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:37.760249    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:37.760514 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:37.760336    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:39.422353 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:39.422322    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzsm\" (UniqueName: \"kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm\") pod \"network-check-target-nx8cd\" (UID: \"0dfc5886-2c51-4274-a603-260a4d3b4688\") " pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:39.422959 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:39.422490    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 16:36:39.422959 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:39.422514    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 16:36:39.422959 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:39.422529    2554 projected.go:196] Error preparing data for projected volume kube-api-access-zlzsm for pod openshift-network-diagnostics/network-check-target-nx8cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 16:36:39.422959 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:39.422590    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm podName:0dfc5886-2c51-4274-a603-260a4d3b4688 nodeName:}" failed. No retries permitted until 2026-07-18 16:36:55.422576594 +0000 UTC m=+34.120407256 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zlzsm" (UniqueName: "kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm") pod "network-check-target-nx8cd" (UID: "0dfc5886-2c51-4274-a603-260a4d3b4688") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 16:36:39.522859 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:39.522809    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:39.523011 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:39.522876    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:39.523011 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:39.522975    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 16:36:39.523118 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:39.523051    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs podName:2f9ff934-e093-4adb-b818-279e624555de nodeName:}" failed. No retries permitted until 2026-07-18 16:36:55.523030528 +0000 UTC m=+34.220861213 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs") pod "network-metrics-daemon-hczmz" (UID: "2f9ff934-e093-4adb-b818-279e624555de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 16:36:39.523118 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:39.522975    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 16:36:39.523203 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:39.523130    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret podName:5a99a42b-8462-4cc8-9512-e9316a46672f nodeName:}" failed. No retries permitted until 2026-07-18 16:36:55.523117789 +0000 UTC m=+34.220948453 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret") pod "global-pull-secret-syncer-wzt7c" (UID: "5a99a42b-8462-4cc8-9512-e9316a46672f") : object "kube-system"/"original-pull-secret" not registered
Jul 18 16:36:39.760521 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:39.760441    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:39.760521 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:39.760466    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:39.760731 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:39.760441    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:39.760731 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:39.760572    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:39.760731 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:39.760635    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:39.760731 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:39.760704    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:41.761544 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:41.761515    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:41.761953 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:41.761614    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:41.761953 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:41.761937    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:41.762074 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:41.762009    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:41.762074 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:41.762054    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:41.762171 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:41.762104    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:43.762984 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:43.762954    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:43.763430 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:43.763080    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:43.763430 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:43.763093    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:43.763430 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:43.763112    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:43.763430 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:43.763184    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:43.763430 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:43.763259    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:44.897366 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.896956    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6" event={"ID":"7d87a70a-de90-4227-8a88-9a20bc99967e","Type":"ContainerStarted","Data":"ba722080ea08e4624974f288f2ba41976f5e4e671b13a3c8887510c9adb05bd7"}
Jul 18 16:36:44.898065 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.897480    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6" event={"ID":"7d87a70a-de90-4227-8a88-9a20bc99967e","Type":"ContainerStarted","Data":"1ad723eef116450934ad8b680e625d1764d1464a123679a037a47863e1be9f24"}
Jul 18 16:36:44.898065 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.897543    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6" event={"ID":"7d87a70a-de90-4227-8a88-9a20bc99967e","Type":"ContainerStarted","Data":"d01f493dec258d2366ce58ee02727797f84fe3c51cd26c4137c21236c1ba88ca"}
Jul 18 16:36:44.898065 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.897592    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6" event={"ID":"7d87a70a-de90-4227-8a88-9a20bc99967e","Type":"ContainerStarted","Data":"bf0089ac4c9382bbb4ea05eac33a402cc8be2d902a1bbe793fc757a91bec68e3"}
Jul 18 16:36:44.898700 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.898681    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5" event={"ID":"01b34321-bf74-4a38-9791-c85a14974eff","Type":"ContainerStarted","Data":"9e41a5b42c339480275a9d54821033f235c995ad1dd182239d9a4644dd391c41"}
Jul 18 16:36:44.899990 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.899965    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6jtwc" event={"ID":"a48a86ca-3a85-46b8-ada3-0125ec0e9546","Type":"ContainerStarted","Data":"d04f98a2d7f55484121a97b0efc42d73b78b4541bd782e3932ed7d0ebdba7f69"}
Jul 18 16:36:44.901377 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.901359    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wfz6t" event={"ID":"ac112e8c-d18e-4f1b-8d1e-8675a88cac3f","Type":"ContainerStarted","Data":"99ceb278f28c1ec564649e20bdf175259318ba607c25246cfeba5f22693af304"}
Jul 18 16:36:44.902636 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.902618    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sz6hk" event={"ID":"8c71dd87-f690-4c8e-b286-6eff3854fd93","Type":"ContainerStarted","Data":"e2923b08b69f7068c3f384d71992fb44bc92952c840cbbae564b572881b25cf2"}
Jul 18 16:36:44.903936 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.903914    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pvzgv" event={"ID":"544a8f18-3d52-422e-8a65-b5b488e82bdd","Type":"ContainerStarted","Data":"78fb0d25d176fcabb7aa8231721c5cd50f0a3f4ab863efb2062d09f827207a78"}
Jul 18 16:36:44.905456 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.905425    2554 generic.go:356] "Generic (PLEG): container finished" podID="087efd0f-84d1-46e1-b971-a5126ec5b252" containerID="164c902f182c14616d8a7ef456f74c1efe11596a1fcebb83dd4594bf6bd2a52a" exitCode=0
Jul 18 16:36:44.905555 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.905508    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ss97s" event={"ID":"087efd0f-84d1-46e1-b971-a5126ec5b252","Type":"ContainerDied","Data":"164c902f182c14616d8a7ef456f74c1efe11596a1fcebb83dd4594bf6bd2a52a"}
Jul 18 16:36:44.906806 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.906733    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kf9nq" event={"ID":"f186e77b-ccfd-461c-8cb8-e4f3a746fbeb","Type":"ContainerStarted","Data":"4fd03f012f3481c5147db439021cd1d404e3663785e84700a6b7775bd24c527c"}
Jul 18 16:36:44.914825 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.914738    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-77.ec2.internal" podStartSLOduration=21.914726873 podStartE2EDuration="21.914726873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 16:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:36:26.870267386 +0000 UTC m=+5.568098075" watchObservedRunningTime="2026-07-18 16:36:44.914726873 +0000 UTC m=+23.612557558"
Jul 18 16:36:44.927708 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.927672    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kf9nq" podStartSLOduration=11.363856721 podStartE2EDuration="22.927658915s" totalImagesPullingTime="11.563802194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:36:44.927618624 +0000 UTC m=+23.625449304" watchObservedRunningTime="2026-07-18 16:36:44.927658915 +0000 UTC m=+23.625489598"
Jul 18 16:36:44.927832 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.927814    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6jtwc" podStartSLOduration=3.259534965 podStartE2EDuration="22.927808531s" totalImagesPullingTime="19.668273566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 16:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:36:44.91448423 +0000 UTC m=+23.612314913" watchObservedRunningTime="2026-07-18 16:36:44.927808531 +0000 UTC m=+23.625639261"
Jul 18 16:36:44.944067 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.944019    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wfz6t" podStartSLOduration=3.272247547 podStartE2EDuration="22.943994999s" totalImagesPullingTime="19.671747452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 16:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:36:44.943329713 +0000 UTC m=+23.641160378" watchObservedRunningTime="2026-07-18 16:36:44.943994999 +0000 UTC m=+23.641825684"
Jul 18 16:36:44.961083 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.961048    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sz6hk" podStartSLOduration=3.292966781 podStartE2EDuration="22.961035988s" totalImagesPullingTime="19.668069207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 16:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:36:44.960817062 +0000 UTC m=+23.658647745" watchObservedRunningTime="2026-07-18 16:36:44.961035988 +0000 UTC m=+23.658866673"
Jul 18 16:36:44.996083 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:44.996043    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pvzgv" podStartSLOduration=3.283946135 podStartE2EDuration="22.996029607s" totalImagesPullingTime="19.712083472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 16:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:36:44.99539868 +0000 UTC m=+23.693229363" watchObservedRunningTime="2026-07-18 16:36:44.996029607 +0000 UTC m=+23.693860291"
Jul 18 16:36:45.760286 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:45.760129    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:45.760286 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:45.760135    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:45.760286 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:45.760248    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:45.760514 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:45.760143    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:45.760514 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:45.760308    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:45.760514 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:45.760400    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:45.914152 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:45.914118    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6" event={"ID":"7d87a70a-de90-4227-8a88-9a20bc99967e","Type":"ContainerStarted","Data":"4692c44237a8faac75801a109f3ba678e9349a238836d523d4c3ed2680e268fc"}
Jul 18 16:36:45.914580 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:45.914200    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6" event={"ID":"7d87a70a-de90-4227-8a88-9a20bc99967e","Type":"ContainerStarted","Data":"5424e8a9f58c7e652b269ba281733551eca2aa2f2facbbd79eafec850b242080"}
Jul 18 16:36:45.916981 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:45.916952    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5" event={"ID":"01b34321-bf74-4a38-9791-c85a14974eff","Type":"ContainerStarted","Data":"d90ff3b024e647df69eb919b6f7bf323b10229b69dc57a120498527709cc8320"}
Jul 18 16:36:45.917125 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:45.917031    2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 18 16:36:45.918617 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:45.918583    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k7dtb" event={"ID":"ce1c94b6-1296-4d2c-bcff-6797174392c3","Type":"ContainerStarted","Data":"138bd616522ec508949f8209a465e04878972e80ed62d084171dc7e027a0fd3b"}
Jul 18 16:36:45.933670 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:45.933617    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k7dtb" podStartSLOduration=4.270799204 podStartE2EDuration="23.933598733s" totalImagesPullingTime="19.662799529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 16:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:36:45.933237622 +0000 UTC m=+24.631068308" watchObservedRunningTime="2026-07-18 16:36:45.933598733 +0000 UTC m=+24.631429418"
Jul 18 16:36:46.739501 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:46.739401    2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-18T16:36:45.91704448Z","UUID":"5eb7348f-44d4-4583-be5a-b4ec5f3b4485","Handler":null,"Name":"","Endpoint":""}
Jul 18 16:36:46.741136 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:46.741117    2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 18 16:36:46.741249 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:46.741150    2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 18 16:36:46.923478 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:46.923441    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5" event={"ID":"01b34321-bf74-4a38-9791-c85a14974eff","Type":"ContainerStarted","Data":"76116bac8836118ebbc6c107fe0fb430f803aa835beba4138ed58584ba651305"}
Jul 18 16:36:46.939490 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:46.939447    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nl4p5" podStartSLOduration=3.26125039 podStartE2EDuration="24.939433878s" totalImagesPullingTime="21.678183488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 16:36:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:36:46.939253568 +0000 UTC m=+25.637084251" watchObservedRunningTime="2026-07-18 16:36:46.939433878 +0000 UTC m=+25.637264562"
Jul 18 16:36:47.760738 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:47.760710    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:47.760958 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:47.760772    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:47.760958 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:47.760776    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:47.760958 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:47.760889    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:47.761121 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:47.760951    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:47.761121 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:47.760998    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:47.929896 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:47.929859    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6" event={"ID":"7d87a70a-de90-4227-8a88-9a20bc99967e","Type":"ContainerStarted","Data":"03cc4386c222aacaff9831927e9b7433885c25c346b06e28723020b0843c990c"}
Jul 18 16:36:49.045491 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:49.045448    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kf9nq"
Jul 18 16:36:49.046110 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:49.046092    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kf9nq"
Jul 18 16:36:49.760121 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:49.759897    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:49.760267 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:49.759897    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:49.760267 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:49.760162    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:49.760267 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:49.759920    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:49.760384 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:49.760224    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:49.760384 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:49.760334    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:49.935359 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:49.935328    2554 generic.go:356] "Generic (PLEG): container finished" podID="087efd0f-84d1-46e1-b971-a5126ec5b252" containerID="4f0c3f0a9387b7e308f61a41355617658c7db783ec0cf99b728161e76724155d" exitCode=0
Jul 18 16:36:49.935519 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:49.935392    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ss97s" event={"ID":"087efd0f-84d1-46e1-b971-a5126ec5b252","Type":"ContainerDied","Data":"4f0c3f0a9387b7e308f61a41355617658c7db783ec0cf99b728161e76724155d"}
Jul 18 16:36:49.939260 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:49.939235    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6" event={"ID":"7d87a70a-de90-4227-8a88-9a20bc99967e","Type":"ContainerStarted","Data":"a39153e4699b1c04a70c938d5eb02b3062579cde73e9ac796542bec294ec9970"}
Jul 18 16:36:49.939543 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:49.939525    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kf9nq"
Jul 18 16:36:49.940049 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:49.940029    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kf9nq"
Jul 18 16:36:49.983010 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:49.982968    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6" podStartSLOduration=8.263248215 podStartE2EDuration="27.982952824s" totalImagesPullingTime="19.719704609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 16:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:36:49.981041474 +0000 UTC m=+28.678872158" watchObservedRunningTime="2026-07-18 16:36:49.982952824 +0000 UTC m=+28.680783508"
Jul 18 16:36:50.942034 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:50.942009    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:50.942770 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:50.942098    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:50.942770 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:50.942141    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:50.956777 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:50.956740    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:50.957333 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:50.957316    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:36:51.553119 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:51.552806    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nx8cd"]
Jul 18 16:36:51.553388 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:51.553296    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:51.553455 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:51.553403    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:51.553513 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:51.553447    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wzt7c"]
Jul 18 16:36:51.553569 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:51.553557    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:51.553673 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:51.553654    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:51.555262 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:51.555244    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hczmz"]
Jul 18 16:36:51.555357 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:51.555324    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:51.555417 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:51.555398    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:51.946064 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:51.946029    2554 generic.go:356] "Generic (PLEG): container finished" podID="087efd0f-84d1-46e1-b971-a5126ec5b252" containerID="34487414aa656b8fb7feff8257ae7a5c650677bff5c77c5cfdf39ea5452db23e" exitCode=0
Jul 18 16:36:51.946553 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:51.946125    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ss97s" event={"ID":"087efd0f-84d1-46e1-b971-a5126ec5b252","Type":"ContainerDied","Data":"34487414aa656b8fb7feff8257ae7a5c650677bff5c77c5cfdf39ea5452db23e"}
Jul 18 16:36:52.950499 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:52.950470    2554 generic.go:356] "Generic (PLEG): container finished" podID="087efd0f-84d1-46e1-b971-a5126ec5b252" containerID="ed61ac0930ea13ef0d6527256e09dce0fae80b00fc60855ef4e435ea98390635" exitCode=0
Jul 18 16:36:52.950837 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:52.950549    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ss97s" event={"ID":"087efd0f-84d1-46e1-b971-a5126ec5b252","Type":"ContainerDied","Data":"ed61ac0930ea13ef0d6527256e09dce0fae80b00fc60855ef4e435ea98390635"}
Jul 18 16:36:53.760260 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:53.760183    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:53.760415 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:53.760191    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:53.760415 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:53.760309    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:53.760415 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:53.760316    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:53.760557 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:53.760414    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:53.760557 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:53.760491    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:55.442496 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:55.442288    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzsm\" (UniqueName: \"kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm\") pod \"network-check-target-nx8cd\" (UID: \"0dfc5886-2c51-4274-a603-260a4d3b4688\") " pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:55.442496 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:55.442481    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 18 16:36:55.442994 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:55.442511    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 18 16:36:55.442994 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:55.442526    2554 projected.go:196] Error preparing data for projected volume kube-api-access-zlzsm for pod openshift-network-diagnostics/network-check-target-nx8cd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 16:36:55.442994 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:55.442593    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm podName:0dfc5886-2c51-4274-a603-260a4d3b4688 nodeName:}" failed. No retries permitted until 2026-07-18 16:37:27.442573277 +0000 UTC m=+66.140403942 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zlzsm" (UniqueName: "kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm") pod "network-check-target-nx8cd" (UID: "0dfc5886-2c51-4274-a603-260a4d3b4688") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 18 16:36:55.543616 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:55.543583    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:55.543779 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:55.543641    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:55.543779 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:55.543751    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 16:36:55.543779 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:55.543758    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 18 16:36:55.543966 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:55.543837    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs podName:2f9ff934-e093-4adb-b818-279e624555de nodeName:}" failed. No retries permitted until 2026-07-18 16:37:27.543816312 +0000 UTC m=+66.241646986 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs") pod "network-metrics-daemon-hczmz" (UID: "2f9ff934-e093-4adb-b818-279e624555de") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 18 16:36:55.543966 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:55.543872    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret podName:5a99a42b-8462-4cc8-9512-e9316a46672f nodeName:}" failed. No retries permitted until 2026-07-18 16:37:27.543861499 +0000 UTC m=+66.241692177 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret") pod "global-pull-secret-syncer-wzt7c" (UID: "5a99a42b-8462-4cc8-9512-e9316a46672f") : object "kube-system"/"original-pull-secret" not registered
Jul 18 16:36:55.760404 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:55.759925    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:55.760404 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:55.760038    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wzt7c" podUID="5a99a42b-8462-4cc8-9512-e9316a46672f"
Jul 18 16:36:55.760404 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:55.760125    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:55.760404 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:55.760243    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:36:55.760732 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:55.760695    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:55.760883 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:55.760780    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nx8cd" podUID="0dfc5886-2c51-4274-a603-260a4d3b4688"
Jul 18 16:36:56.215296 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:56.215223    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:36:56.313356 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:56.313324    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:36:57.182165 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.182085    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-77.ec2.internal" event="NodeReady"
Jul 18 16:36:57.182566 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.182228    2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 18 16:36:57.229017 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.228974    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-994h4"]
Jul 18 16:36:57.231978 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.231956    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-psptl"]
Jul 18 16:36:57.232123 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.232074    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-994h4"
Jul 18 16:36:57.233546 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.233528    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-psptl"
Jul 18 16:36:57.234884 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.234866    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2jrkz\""
Jul 18 16:36:57.234989 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.234955    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 18 16:36:57.235048 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.234866    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 18 16:36:57.235702 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.235683    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5r7jb\""
Jul 18 16:36:57.235799 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.235748    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 18 16:36:57.236010 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.235986    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 18 16:36:57.236085 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.236023    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 18 16:36:57.241474 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.241455    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-994h4"]
Jul 18 16:36:57.245594 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.245549    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-psptl"]
Jul 18 16:36:57.357102 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.357062    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbd0e94-6c81-4eaa-abb9-522600419f63-config-volume\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:36:57.357269 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.357110    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p7w2\" (UniqueName: \"kubernetes.io/projected/9bbd0e94-6c81-4eaa-abb9-522600419f63-kube-api-access-9p7w2\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:36:57.357269 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.357139    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09878619-4f85-4acb-bce3-94a409e582d1-cert\") pod \"ingress-canary-psptl\" (UID: \"09878619-4f85-4acb-bce3-94a409e582d1\") " pod="openshift-ingress-canary/ingress-canary-psptl"
Jul 18 16:36:57.357269 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.357182    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:36:57.357269 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.357208    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bbd0e94-6c81-4eaa-abb9-522600419f63-tmp-dir\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:36:57.357269 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.357231    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76xvq\" (UniqueName: \"kubernetes.io/projected/09878619-4f85-4acb-bce3-94a409e582d1-kube-api-access-76xvq\") pod \"ingress-canary-psptl\" (UID: \"09878619-4f85-4acb-bce3-94a409e582d1\") " pod="openshift-ingress-canary/ingress-canary-psptl"
Jul 18 16:36:57.358730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.358704    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:36:57.456218 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.456131    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:36:57.458029 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.458006    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbd0e94-6c81-4eaa-abb9-522600419f63-config-volume\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:36:57.458152 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.458046    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p7w2\" (UniqueName: \"kubernetes.io/projected/9bbd0e94-6c81-4eaa-abb9-522600419f63-kube-api-access-9p7w2\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:36:57.458152 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.458072    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09878619-4f85-4acb-bce3-94a409e582d1-cert\") pod \"ingress-canary-psptl\" (UID: \"09878619-4f85-4acb-bce3-94a409e582d1\") " pod="openshift-ingress-canary/ingress-canary-psptl"
Jul 18 16:36:57.458152 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.458126    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:36:57.458298 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.458152    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bbd0e94-6c81-4eaa-abb9-522600419f63-tmp-dir\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:36:57.458298 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.458180    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76xvq\" (UniqueName: \"kubernetes.io/projected/09878619-4f85-4acb-bce3-94a409e582d1-kube-api-access-76xvq\") pod \"ingress-canary-psptl\" (UID: \"09878619-4f85-4acb-bce3-94a409e582d1\") " pod="openshift-ingress-canary/ingress-canary-psptl"
Jul 18 16:36:57.458298 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:57.458235    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 18 16:36:57.458443 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:57.458302    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls podName:9bbd0e94-6c81-4eaa-abb9-522600419f63 nodeName:}" failed. No retries permitted until 2026-07-18 16:36:57.958280913 +0000 UTC m=+36.656111588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls") pod "dns-default-994h4" (UID: "9bbd0e94-6c81-4eaa-abb9-522600419f63") : secret "dns-default-metrics-tls" not found
Jul 18 16:36:57.458510 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.458455    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bbd0e94-6c81-4eaa-abb9-522600419f63-tmp-dir\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:36:57.458642 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.458620    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbd0e94-6c81-4eaa-abb9-522600419f63-config-volume\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:36:57.463025 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.463008    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09878619-4f85-4acb-bce3-94a409e582d1-cert\") pod \"ingress-canary-psptl\" (UID: \"09878619-4f85-4acb-bce3-94a409e582d1\") " pod="openshift-ingress-canary/ingress-canary-psptl"
Jul 18 16:36:57.470404 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.470380    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p7w2\" (UniqueName: \"kubernetes.io/projected/9bbd0e94-6c81-4eaa-abb9-522600419f63-kube-api-access-9p7w2\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:36:57.470738 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.470716    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76xvq\" (UniqueName: \"kubernetes.io/projected/09878619-4f85-4acb-bce3-94a409e582d1-kube-api-access-76xvq\") pod \"ingress-canary-psptl\" (UID: \"09878619-4f85-4acb-bce3-94a409e582d1\") " pod="openshift-ingress-canary/ingress-canary-psptl"
Jul 18 16:36:57.520182 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.520131    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:36:57.551062 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.551039    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-psptl"
Jul 18 16:36:57.586397 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.586357    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:36:57.691372 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.691184    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-psptl"]
Jul 18 16:36:57.694834 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:36:57.694807    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09878619_4f85_4acb_bce3_94a409e582d1.slice/crio-e9685fcb89feebb8467d5006d2199eadd7e41ee7255462c8f0077c2bb58c34e3 WatchSource:0}: Error finding container e9685fcb89feebb8467d5006d2199eadd7e41ee7255462c8f0077c2bb58c34e3: Status 404 returned error can't find the container with id e9685fcb89feebb8467d5006d2199eadd7e41ee7255462c8f0077c2bb58c34e3
Jul 18 16:36:57.760591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.760552    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:36:57.760765 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.760552    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:36:57.760829 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.760556    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:36:57.764516 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.764495    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n9vx5\""
Jul 18 16:36:57.764623 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.764534    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 18 16:36:57.764759 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.764744    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 18 16:36:57.764838 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.764822    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 18 16:36:57.764917 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.764871    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 18 16:36:57.765171 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.765151    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pm4pn\""
Jul 18 16:36:57.962031 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.961950    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:36:57.962171 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:57.962102    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 18 16:36:57.962171 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:57.962167    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls podName:9bbd0e94-6c81-4eaa-abb9-522600419f63 nodeName:}" failed. No retries permitted until 2026-07-18 16:36:58.962146894 +0000 UTC m=+37.659977556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls") pod "dns-default-994h4" (UID: "9bbd0e94-6c81-4eaa-abb9-522600419f63") : secret "dns-default-metrics-tls" not found
Jul 18 16:36:57.964769 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.964746    2554 generic.go:356] "Generic (PLEG): container finished" podID="087efd0f-84d1-46e1-b971-a5126ec5b252" containerID="1caeb0c2870802d7a69f10cee92e965bb7a584c23f5f4fd9e84cc06e4600b9fb" exitCode=0
Jul 18 16:36:57.964927 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.964829    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ss97s" event={"ID":"087efd0f-84d1-46e1-b971-a5126ec5b252","Type":"ContainerDied","Data":"1caeb0c2870802d7a69f10cee92e965bb7a584c23f5f4fd9e84cc06e4600b9fb"}
Jul 18 16:36:57.966040 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:57.966023    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-psptl" event={"ID":"09878619-4f85-4acb-bce3-94a409e582d1","Type":"ContainerStarted","Data":"e9685fcb89feebb8467d5006d2199eadd7e41ee7255462c8f0077c2bb58c34e3"}
Jul 18 16:36:58.437381 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:58.437296    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:36:58.967325 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:58.967292    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:36:58.967486 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:58.967415    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 18 16:36:58.967486 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:36:58.967472    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls podName:9bbd0e94-6c81-4eaa-abb9-522600419f63 nodeName:}" failed. No retries permitted until 2026-07-18 16:37:00.967456394 +0000 UTC m=+39.665287077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls") pod "dns-default-994h4" (UID: "9bbd0e94-6c81-4eaa-abb9-522600419f63") : secret "dns-default-metrics-tls" not found
Jul 18 16:36:58.971740 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:58.971713    2554 generic.go:356] "Generic (PLEG): container finished" podID="087efd0f-84d1-46e1-b971-a5126ec5b252" containerID="bad159efbb7524afe59688591499ef2b1715b43528045efd561a32c27273f4f2" exitCode=0
Jul 18 16:36:58.971864 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:58.971764    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ss97s" event={"ID":"087efd0f-84d1-46e1-b971-a5126ec5b252","Type":"ContainerDied","Data":"bad159efbb7524afe59688591499ef2b1715b43528045efd561a32c27273f4f2"}
Jul 18 16:36:59.514017 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:59.513984    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:36:59.614074 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:59.614041    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:36:59.978805 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:36:59.978223    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ss97s" event={"ID":"087efd0f-84d1-46e1-b971-a5126ec5b252","Type":"ContainerStarted","Data":"e4502ef7f5902d010c0112decfabd8bcc0f5c9ee8e7bf17f4408591f67e512eb"}
Jul 18 16:37:00.007020 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:00.006968    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ss97s" podStartSLOduration=7.371026442 podStartE2EDuration="38.00694993s" totalImagesPullingTime="30.635923488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 16:36:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:37:00.006484156 +0000 UTC m=+38.704314841" watchObservedRunningTime="2026-07-18 16:37:00.00694993 +0000 UTC m=+38.704780615"
Jul 18 16:37:00.013077 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:00.013049    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:00.073110 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:00.073071    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:00.461288 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:00.461211    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:00.559115 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:00.559066    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:00.658617 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:00.658585    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:00.862177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:00.862146    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:00.911246 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:00.911217    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:00.981433 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:00.981398    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:37:00.981574 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:37:00.981559    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 18 16:37:00.981650 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:37:00.981630    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls podName:9bbd0e94-6c81-4eaa-abb9-522600419f63 nodeName:}" failed. No retries permitted until 2026-07-18 16:37:04.981609134 +0000 UTC m=+43.679439813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls") pod "dns-default-994h4" (UID: "9bbd0e94-6c81-4eaa-abb9-522600419f63") : secret "dns-default-metrics-tls" not found
Jul 18 16:37:01.887955 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:01.887917    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:01.987083 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:01.987040    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-psptl" event={"ID":"09878619-4f85-4acb-bce3-94a409e582d1","Type":"ContainerStarted","Data":"29d6514550d6485625214ea0d19ac9abc455f056989d04ce647ef998a6b778d5"}
Jul 18 16:37:01.987649 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:01.987627    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:02.003008 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:02.002965    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-psptl" podStartSLOduration=1.8569525260000002 podStartE2EDuration="5.002952224s" totalImagesPullingTime="3.145999698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:36:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:37:02.002559384 +0000 UTC m=+40.700390069" watchObservedRunningTime="2026-07-18 16:37:02.002952224 +0000 UTC m=+40.700782915"
Jul 18 16:37:05.010777 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:05.010741    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:37:05.011174 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:37:05.010864    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 18 16:37:05.011174 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:37:05.010921    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls podName:9bbd0e94-6c81-4eaa-abb9-522600419f63 nodeName:}" failed. No retries permitted until 2026-07-18 16:37:13.010907272 +0000 UTC m=+51.708737934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls") pod "dns-default-994h4" (UID: "9bbd0e94-6c81-4eaa-abb9-522600419f63") : secret "dns-default-metrics-tls" not found
Jul 18 16:37:09.904738 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:09.904703    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:09.955458 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:09.955414    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:12.600904 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:12.600872    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:12.700788 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:12.700756    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:13.063354 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:13.063253    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:37:13.063498 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:37:13.063371    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 18 16:37:13.063498 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:37:13.063432    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls podName:9bbd0e94-6c81-4eaa-abb9-522600419f63 nodeName:}" failed. No retries permitted until 2026-07-18 16:37:29.063417314 +0000 UTC m=+67.761247975 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls") pod "dns-default-994h4" (UID: "9bbd0e94-6c81-4eaa-abb9-522600419f63") : secret "dns-default-metrics-tls" not found
Jul 18 16:37:15.044924 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:15.044888    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:15.149389 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:15.149353    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:22.025955 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:22.025919    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:22.090866 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:22.090793    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:22.970398 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:22.970371    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hzpm6"
Jul 18 16:37:23.137188 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:23.137152    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:23.214272 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:23.214243    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:27.461353 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:27.461305    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzsm\" (UniqueName: \"kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm\") pod \"network-check-target-nx8cd\" (UID: \"0dfc5886-2c51-4274-a603-260a4d3b4688\") " pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:37:27.464590 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:27.464572    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 18 16:37:27.475704 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:27.475680    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 18 16:37:27.486968 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:27.486950    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlzsm\" (UniqueName: \"kubernetes.io/projected/0dfc5886-2c51-4274-a603-260a4d3b4688-kube-api-access-zlzsm\") pod \"network-check-target-nx8cd\" (UID: \"0dfc5886-2c51-4274-a603-260a4d3b4688\") " pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:37:27.562418 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:27.562379    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:37:27.562582 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:27.562435    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:37:27.565487 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:27.565464    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 18 16:37:27.565580 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:27.565561    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 18 16:37:27.572795 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:37:27.572775    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 18 16:37:27.572888 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:37:27.572877    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs podName:2f9ff934-e093-4adb-b818-279e624555de nodeName:}" failed. No retries permitted until 2026-07-18 16:38:31.572831931 +0000 UTC m=+130.270662593 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs") pod "network-metrics-daemon-hczmz" (UID: "2f9ff934-e093-4adb-b818-279e624555de") : secret "metrics-daemon-secret" not found
Jul 18 16:37:27.575093 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:27.575075    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a99a42b-8462-4cc8-9512-e9316a46672f-original-pull-secret\") pod \"global-pull-secret-syncer-wzt7c\" (UID: \"5a99a42b-8462-4cc8-9512-e9316a46672f\") " pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:37:27.774974 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:27.774903    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n9vx5\""
Jul 18 16:37:27.782909 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:27.782888    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:37:27.832556 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:27.832529    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wzt7c"
Jul 18 16:37:27.910065 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:27.910033    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nx8cd"]
Jul 18 16:37:27.916036 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:37:27.916002    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dfc5886_2c51_4274_a603_260a4d3b4688.slice/crio-53c46942d9d153d218e0203aeb70143ff205fb932dadee068ae81ee71b902c8e WatchSource:0}: Error finding container 53c46942d9d153d218e0203aeb70143ff205fb932dadee068ae81ee71b902c8e: Status 404 returned error can't find the container with id 53c46942d9d153d218e0203aeb70143ff205fb932dadee068ae81ee71b902c8e
Jul 18 16:37:27.956670 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:27.956643    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wzt7c"]
Jul 18 16:37:27.963556 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:37:27.960884    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a99a42b_8462_4cc8_9512_e9316a46672f.slice/crio-e34dac7d28ebd5f8e4379b52e4250d976b5c3043ee3f131743d8e9217d797411 WatchSource:0}: Error finding container e34dac7d28ebd5f8e4379b52e4250d976b5c3043ee3f131743d8e9217d797411: Status 404 returned error can't find the container with id e34dac7d28ebd5f8e4379b52e4250d976b5c3043ee3f131743d8e9217d797411
Jul 18 16:37:28.045469 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:28.045376    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nx8cd" event={"ID":"0dfc5886-2c51-4274-a603-260a4d3b4688","Type":"ContainerStarted","Data":"53c46942d9d153d218e0203aeb70143ff205fb932dadee068ae81ee71b902c8e"}
Jul 18 16:37:28.046429 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:28.046402    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wzt7c" event={"ID":"5a99a42b-8462-4cc8-9512-e9316a46672f","Type":"ContainerStarted","Data":"e34dac7d28ebd5f8e4379b52e4250d976b5c3043ee3f131743d8e9217d797411"}
Jul 18 16:37:29.071903 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:29.071863    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:37:29.072499 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:37:29.072062    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 18 16:37:29.072499 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:37:29.072135    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls podName:9bbd0e94-6c81-4eaa-abb9-522600419f63 nodeName:}" failed. No retries permitted until 2026-07-18 16:38:01.072115859 +0000 UTC m=+99.769946523 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls") pod "dns-default-994h4" (UID: "9bbd0e94-6c81-4eaa-abb9-522600419f63") : secret "dns-default-metrics-tls" not found
Jul 18 16:37:33.062762 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:33.062730    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nx8cd" event={"ID":"0dfc5886-2c51-4274-a603-260a4d3b4688","Type":"ContainerStarted","Data":"b3653954a49db5478c8a4a51db83b99e3c0883ba17fb28de4d60ee6ab8e7745b"}
Jul 18 16:37:33.063262 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:33.062882    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:37:33.064175 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:33.064153    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wzt7c" event={"ID":"5a99a42b-8462-4cc8-9512-e9316a46672f","Type":"ContainerStarted","Data":"3146f636e705ced56bd8d9b393e5a3d2cb3642a5a73221e2f7a3aed99a086da9"}
Jul 18 16:37:33.080249 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:33.080209    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nx8cd" podStartSLOduration=66.610658597 podStartE2EDuration="1m11.080193295s" totalImagesPullingTime="4.469534698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:37:33.080007199 +0000 UTC m=+71.777837880" watchObservedRunningTime="2026-07-18 16:37:33.080193295 +0000 UTC m=+71.778023980"
Jul 18 16:37:33.097166 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:33.097123    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wzt7c" podStartSLOduration=66.66926124 podStartE2EDuration="1m11.097109605s" totalImagesPullingTime="4.427848365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 16:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:37:33.096243376 +0000 UTC m=+71.794074059" watchObservedRunningTime="2026-07-18 16:37:33.097109605 +0000 UTC m=+71.794940288"
Jul 18 16:37:34.585102 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:34.585059    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:34.655922 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:34.655893    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:59.619265 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:59.619154    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:37:59.695264 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:37:59.695238    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:38:01.080141 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:01.080094    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:38:01.080481 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:38:01.080257    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 18 16:38:01.080481 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:38:01.080329    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls podName:9bbd0e94-6c81-4eaa-abb9-522600419f63 nodeName:}" failed. No retries permitted until 2026-07-18 16:39:05.080313071 +0000 UTC m=+163.778143736 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls") pod "dns-default-994h4" (UID: "9bbd0e94-6c81-4eaa-abb9-522600419f63") : secret "dns-default-metrics-tls" not found
Jul 18 16:38:04.069114 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:04.069084    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nx8cd"
Jul 18 16:38:19.914307 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:19.914274    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"]
Jul 18 16:38:19.915932 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:19.915915    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"
Jul 18 16:38:19.918330 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:19.918308    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 18 16:38:19.918677 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:19.918660    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 18 16:38:19.919609 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:19.919594    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 18 16:38:19.919685 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:19.919597    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 18 16:38:19.925754 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:19.925734    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"]
Jul 18 16:38:19.999523 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:19.999486    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgk4l\" (UniqueName: \"kubernetes.io/projected/2898d165-2009-4ecf-b751-607b5bbe84eb-kube-api-access-xgk4l\") pod \"klusterlet-addon-workmgr-c76fb848-2cpk5\" (UID: \"2898d165-2009-4ecf-b751-607b5bbe84eb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"
Jul 18 16:38:19.999686 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:19.999541    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2898d165-2009-4ecf-b751-607b5bbe84eb-klusterlet-config\") pod \"klusterlet-addon-workmgr-c76fb848-2cpk5\" (UID: \"2898d165-2009-4ecf-b751-607b5bbe84eb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"
Jul 18 16:38:19.999686 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:19.999570    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2898d165-2009-4ecf-b751-607b5bbe84eb-tmp\") pod \"klusterlet-addon-workmgr-c76fb848-2cpk5\" (UID: \"2898d165-2009-4ecf-b751-607b5bbe84eb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"
Jul 18 16:38:20.100524 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:20.100487    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgk4l\" (UniqueName: \"kubernetes.io/projected/2898d165-2009-4ecf-b751-607b5bbe84eb-kube-api-access-xgk4l\") pod \"klusterlet-addon-workmgr-c76fb848-2cpk5\" (UID: \"2898d165-2009-4ecf-b751-607b5bbe84eb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"
Jul 18 16:38:20.100699 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:20.100538    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2898d165-2009-4ecf-b751-607b5bbe84eb-klusterlet-config\") pod \"klusterlet-addon-workmgr-c76fb848-2cpk5\" (UID: \"2898d165-2009-4ecf-b751-607b5bbe84eb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"
Jul 18 16:38:20.100699 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:20.100571    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2898d165-2009-4ecf-b751-607b5bbe84eb-tmp\") pod \"klusterlet-addon-workmgr-c76fb848-2cpk5\" (UID: \"2898d165-2009-4ecf-b751-607b5bbe84eb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"
Jul 18 16:38:20.100982 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:20.100965    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2898d165-2009-4ecf-b751-607b5bbe84eb-tmp\") pod \"klusterlet-addon-workmgr-c76fb848-2cpk5\" (UID: \"2898d165-2009-4ecf-b751-607b5bbe84eb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"
Jul 18 16:38:20.104129 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:20.104108    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2898d165-2009-4ecf-b751-607b5bbe84eb-klusterlet-config\") pod \"klusterlet-addon-workmgr-c76fb848-2cpk5\" (UID: \"2898d165-2009-4ecf-b751-607b5bbe84eb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"
Jul 18 16:38:20.110792 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:20.110768    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgk4l\" (UniqueName: \"kubernetes.io/projected/2898d165-2009-4ecf-b751-607b5bbe84eb-kube-api-access-xgk4l\") pod \"klusterlet-addon-workmgr-c76fb848-2cpk5\" (UID: \"2898d165-2009-4ecf-b751-607b5bbe84eb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"
Jul 18 16:38:20.224771 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:20.224693    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"
Jul 18 16:38:20.342081 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:20.342054    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"]
Jul 18 16:38:20.344953 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:38:20.344923    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2898d165_2009_4ecf_b751_607b5bbe84eb.slice/crio-9e66b307c23127dcf223de8419ef6c9f989c31c35c6ecef5d46b251941a089e0 WatchSource:0}: Error finding container 9e66b307c23127dcf223de8419ef6c9f989c31c35c6ecef5d46b251941a089e0: Status 404 returned error can't find the container with id 9e66b307c23127dcf223de8419ef6c9f989c31c35c6ecef5d46b251941a089e0
Jul 18 16:38:21.180990 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:21.180950    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5" event={"ID":"2898d165-2009-4ecf-b751-607b5bbe84eb","Type":"ContainerStarted","Data":"9e66b307c23127dcf223de8419ef6c9f989c31c35c6ecef5d46b251941a089e0"}
Jul 18 16:38:24.189159 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:24.189112    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5" event={"ID":"2898d165-2009-4ecf-b751-607b5bbe84eb","Type":"ContainerStarted","Data":"36a23ba99d83e07cbcef7a563646a30ef025a049c926082c343fd77e36a74481"}
Jul 18 16:38:24.189553 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:24.189332    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"
Jul 18 16:38:24.190614 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:24.190593    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"
Jul 18 16:38:24.204511 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:24.204462    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5" podStartSLOduration=1.59123854 podStartE2EDuration="5.204447207s" totalImagesPullingTime="3.613208667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:38:24.20325265 +0000 UTC m=+122.901083334" watchObservedRunningTime="2026-07-18 16:38:24.204447207 +0000 UTC m=+122.902277890"
Jul 18 16:38:31.586583 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:38:31.586533    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:38:31.587013 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:38:31.586692    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 18 16:38:31.587013 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:38:31.586761    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs podName:2f9ff934-e093-4adb-b818-279e624555de nodeName:}" failed. No retries permitted until 2026-07-18 16:40:33.586744536 +0000 UTC m=+252.284575214 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs") pod "network-metrics-daemon-hczmz" (UID: "2f9ff934-e093-4adb-b818-279e624555de") : secret "metrics-daemon-secret" not found
Jul 18 16:39:00.244301 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:39:00.244259    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-994h4" podUID="9bbd0e94-6c81-4eaa-abb9-522600419f63"
Jul 18 16:39:00.271036 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:39:00.271008    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-994h4"
Jul 18 16:39:00.837118 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:39:00.837072    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:39:05.112719 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:39:05.112681    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:39:05.113134 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:39:05.112831    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 18 16:39:05.113134 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:39:05.112911    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls podName:9bbd0e94-6c81-4eaa-abb9-522600419f63 nodeName:}" failed. No retries permitted until 2026-07-18 16:41:07.112896814 +0000 UTC m=+285.810727476 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls") pod "dns-default-994h4" (UID: "9bbd0e94-6c81-4eaa-abb9-522600419f63") : secret "dns-default-metrics-tls" not found
Jul 18 16:39:12.759922 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:39:12.759872    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:40:08.033158 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:08.033122    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:09.885171 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:09.885138    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:24.189860 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:24.189793    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5" podUID="2898d165-2009-4ecf-b751-607b5bbe84eb" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused"
Jul 18 16:40:24.470348 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:24.470315    2554 generic.go:356] "Generic (PLEG): container finished" podID="2898d165-2009-4ecf-b751-607b5bbe84eb" containerID="36a23ba99d83e07cbcef7a563646a30ef025a049c926082c343fd77e36a74481" exitCode=1
Jul 18 16:40:24.470496 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:24.470389    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5" event={"ID":"2898d165-2009-4ecf-b751-607b5bbe84eb","Type":"ContainerDied","Data":"36a23ba99d83e07cbcef7a563646a30ef025a049c926082c343fd77e36a74481"}
Jul 18 16:40:24.470686 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:24.470674    2554 scope.go:117] "RemoveContainer" containerID="36a23ba99d83e07cbcef7a563646a30ef025a049c926082c343fd77e36a74481"
Jul 18 16:40:25.474911 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:25.474879    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5" event={"ID":"2898d165-2009-4ecf-b751-607b5bbe84eb","Type":"ContainerStarted","Data":"9e9e4ebf40ebe78d8e19841b08e6ffdbbe792fceaa58a7c4d601c707cc3f9fe8"}
Jul 18 16:40:25.475291 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:25.475170    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"
Jul 18 16:40:25.475744 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:25.475725    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c76fb848-2cpk5"
Jul 18 16:40:32.274194 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.274157    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-bq5fh"]
Jul 18 16:40:32.276887 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.276871    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-bq5fh"
Jul 18 16:40:32.279308 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.279288    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dqjtx\""
Jul 18 16:40:32.282123 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.282097    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6947f4bd9f-px842"]
Jul 18 16:40:32.284698 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.284681    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.289429 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.289406    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 18 16:40:32.289559 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.289536    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6kmj4\""
Jul 18 16:40:32.289664 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.289438    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 18 16:40:32.289664 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.289515    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 18 16:40:32.291209 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.291164    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-bq5fh"]
Jul 18 16:40:32.294683 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.294569    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 18 16:40:32.303909 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.303836    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6947f4bd9f-px842"]
Jul 18 16:40:32.389938 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.389907    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-kvsvt"]
Jul 18 16:40:32.392726 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.392705    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.395180 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.395164    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 18 16:40:32.395471 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.395458    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-h92db\""
Jul 18 16:40:32.395548 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.395494    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 18 16:40:32.395598 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.395550    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 18 16:40:32.396053 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.396037    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 18 16:40:32.403437 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.403421    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 18 16:40:32.405609 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.405587    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-kvsvt"]
Jul 18 16:40:32.409018 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.408993    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chzcq\" (UniqueName: \"kubernetes.io/projected/efa4d0b3-fd00-42ad-909b-fe2894c3adb3-kube-api-access-chzcq\") pod \"network-check-source-7bc5b6bf85-bq5fh\" (UID: \"efa4d0b3-fd00-42ad-909b-fe2894c3adb3\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-bq5fh"
Jul 18 16:40:32.409123 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.409031    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-bound-sa-token\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.409123 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.409069    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.409123 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.409095    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-certificates\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.409291 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.409179    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26cc0844-d905-444b-bea5-d6b4a2b148a2-image-registry-private-configuration\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.409291 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.409260    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26cc0844-d905-444b-bea5-d6b4a2b148a2-installation-pull-secrets\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.409376 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.409291    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26cc0844-d905-444b-bea5-d6b4a2b148a2-ca-trust-extracted\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.409376 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.409317    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcnjv\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-kube-api-access-lcnjv\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.409376 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.409344    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26cc0844-d905-444b-bea5-d6b4a2b148a2-trusted-ca\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.510041 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510009    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-certificates\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.510197 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510048    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26cc0844-d905-444b-bea5-d6b4a2b148a2-image-registry-private-configuration\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.510275 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510259    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05af46b3-3c72-416b-b837-d9412910c1b5-service-ca-bundle\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.510332 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510318    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26cc0844-d905-444b-bea5-d6b4a2b148a2-installation-pull-secrets\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.510386 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510350    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26cc0844-d905-444b-bea5-d6b4a2b148a2-ca-trust-extracted\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.510386 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510376    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05af46b3-3c72-416b-b837-d9412910c1b5-tmp\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.510482 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510408    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcnjv\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-kube-api-access-lcnjv\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.510482 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510438    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26cc0844-d905-444b-bea5-d6b4a2b148a2-trusted-ca\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.510575 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510479    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzrlz\" (UniqueName: \"kubernetes.io/projected/05af46b3-3c72-416b-b837-d9412910c1b5-kube-api-access-lzrlz\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.510575 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510517    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/05af46b3-3c72-416b-b837-d9412910c1b5-snapshots\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.510575 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510541    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05af46b3-3c72-416b-b837-d9412910c1b5-serving-cert\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.510718 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510619    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chzcq\" (UniqueName: \"kubernetes.io/projected/efa4d0b3-fd00-42ad-909b-fe2894c3adb3-kube-api-access-chzcq\") pod \"network-check-source-7bc5b6bf85-bq5fh\" (UID: \"efa4d0b3-fd00-42ad-909b-fe2894c3adb3\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-bq5fh"
Jul 18 16:40:32.510833 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510807    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-bound-sa-token\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.511123 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510872    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05af46b3-3c72-416b-b837-d9412910c1b5-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.511123 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510739    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26cc0844-d905-444b-bea5-d6b4a2b148a2-ca-trust-extracted\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.511123 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510682    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-certificates\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.511123 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.510934    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.511123 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:32.511057    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 18 16:40:32.511123 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:32.511072    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6947f4bd9f-px842: secret "image-registry-tls" not found
Jul 18 16:40:32.511123 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:32.511133    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls podName:26cc0844-d905-444b-bea5-d6b4a2b148a2 nodeName:}" failed. No retries permitted until 2026-07-18 16:40:33.011115733 +0000 UTC m=+251.708946395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls") pod "image-registry-6947f4bd9f-px842" (UID: "26cc0844-d905-444b-bea5-d6b4a2b148a2") : secret "image-registry-tls" not found
Jul 18 16:40:32.511653 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.511491    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26cc0844-d905-444b-bea5-d6b4a2b148a2-trusted-ca\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.512696 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.512670    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26cc0844-d905-444b-bea5-d6b4a2b148a2-image-registry-private-configuration\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.512874 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.512840    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26cc0844-d905-444b-bea5-d6b4a2b148a2-installation-pull-secrets\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.519315 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.519298    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chzcq\" (UniqueName: \"kubernetes.io/projected/efa4d0b3-fd00-42ad-909b-fe2894c3adb3-kube-api-access-chzcq\") pod \"network-check-source-7bc5b6bf85-bq5fh\" (UID: \"efa4d0b3-fd00-42ad-909b-fe2894c3adb3\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-bq5fh"
Jul 18 16:40:32.519798 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.519782    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-bound-sa-token\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.520191 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.520171    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcnjv\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-kube-api-access-lcnjv\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:32.521960 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.521918    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:32.579132 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.579075    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:32.589719 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.589698    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-bq5fh"
Jul 18 16:40:32.611962 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.611938    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05af46b3-3c72-416b-b837-d9412910c1b5-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.612080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.612001    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05af46b3-3c72-416b-b837-d9412910c1b5-service-ca-bundle\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.612080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.612053    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05af46b3-3c72-416b-b837-d9412910c1b5-tmp\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.612196 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.612091    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrlz\" (UniqueName: \"kubernetes.io/projected/05af46b3-3c72-416b-b837-d9412910c1b5-kube-api-access-lzrlz\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.612196 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.612128    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/05af46b3-3c72-416b-b837-d9412910c1b5-snapshots\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.612196 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.612153    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05af46b3-3c72-416b-b837-d9412910c1b5-serving-cert\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.612479 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.612453    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05af46b3-3c72-416b-b837-d9412910c1b5-tmp\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.612648 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.612609    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05af46b3-3c72-416b-b837-d9412910c1b5-service-ca-bundle\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.612817 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.612785    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/05af46b3-3c72-416b-b837-d9412910c1b5-snapshots\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.612817 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.612796    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05af46b3-3c72-416b-b837-d9412910c1b5-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.614715 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.614677    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05af46b3-3c72-416b-b837-d9412910c1b5-serving-cert\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.620710 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.620689    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzrlz\" (UniqueName: \"kubernetes.io/projected/05af46b3-3c72-416b-b837-d9412910c1b5-kube-api-access-lzrlz\") pod \"insights-operator-555d9d7885-kvsvt\" (UID: \"05af46b3-3c72-416b-b837-d9412910c1b5\") " pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.701455 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.701427    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-kvsvt"
Jul 18 16:40:32.713814 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.713792    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-bq5fh"]
Jul 18 16:40:32.717859 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:40:32.717820    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa4d0b3_fd00_42ad_909b_fe2894c3adb3.slice/crio-86688dc68ee797194888e382d1b7629693afb26cc01f80037c25d65d3d62b4ca WatchSource:0}: Error finding container 86688dc68ee797194888e382d1b7629693afb26cc01f80037c25d65d3d62b4ca: Status 404 returned error can't find the container with id 86688dc68ee797194888e382d1b7629693afb26cc01f80037c25d65d3d62b4ca
Jul 18 16:40:32.830889 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:32.830721    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-kvsvt"]
Jul 18 16:40:32.833393 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:40:32.833366    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05af46b3_3c72_416b_b837_d9412910c1b5.slice/crio-cc9ef2a4c1db461222b1eb6f5b719581630052d1e10c93f7bee4114816bb20d3 WatchSource:0}: Error finding container cc9ef2a4c1db461222b1eb6f5b719581630052d1e10c93f7bee4114816bb20d3: Status 404 returned error can't find the container with id cc9ef2a4c1db461222b1eb6f5b719581630052d1e10c93f7bee4114816bb20d3
Jul 18 16:40:33.016061 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:33.016025    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:33.016232 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:33.016184    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 18 16:40:33.016232 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:33.016206    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6947f4bd9f-px842: secret "image-registry-tls" not found
Jul 18 16:40:33.016307 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:33.016260    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls podName:26cc0844-d905-444b-bea5-d6b4a2b148a2 nodeName:}" failed. No retries permitted until 2026-07-18 16:40:34.016244205 +0000 UTC m=+252.714074868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls") pod "image-registry-6947f4bd9f-px842" (UID: "26cc0844-d905-444b-bea5-d6b4a2b148a2") : secret "image-registry-tls" not found
Jul 18 16:40:33.500587 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:33.500516    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kvsvt" event={"ID":"05af46b3-3c72-416b-b837-d9412910c1b5","Type":"ContainerStarted","Data":"cc9ef2a4c1db461222b1eb6f5b719581630052d1e10c93f7bee4114816bb20d3"}
Jul 18 16:40:33.502205 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:33.502175    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-bq5fh" event={"ID":"efa4d0b3-fd00-42ad-909b-fe2894c3adb3","Type":"ContainerStarted","Data":"b9d85cddbb738d0f19a530f4abd01beaac1f4d833b6177205be8289021fda381"}
Jul 18 16:40:33.502333 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:33.502218    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-bq5fh" event={"ID":"efa4d0b3-fd00-42ad-909b-fe2894c3adb3","Type":"ContainerStarted","Data":"86688dc68ee797194888e382d1b7629693afb26cc01f80037c25d65d3d62b4ca"}
Jul 18 16:40:33.516736 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:33.516687    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-bq5fh" podStartSLOduration=1.5166678770000002 podStartE2EDuration="1.516667877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:40:33.516223621 +0000 UTC m=+252.214054306" watchObservedRunningTime="2026-07-18 16:40:33.516667877 +0000 UTC m=+252.214498562"
Jul 18 16:40:33.621419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:33.621380    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:40:33.621649 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:33.621515    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 18 16:40:33.621649 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:33.621621    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs podName:2f9ff934-e093-4adb-b818-279e624555de nodeName:}" failed. No retries permitted until 2026-07-18 16:42:35.621601921 +0000 UTC m=+374.319432586 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs") pod "network-metrics-daemon-hczmz" (UID: "2f9ff934-e093-4adb-b818-279e624555de") : secret "metrics-daemon-secret" not found
Jul 18 16:40:34.025244 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:34.025209    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:34.025437 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:34.025339    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 18 16:40:34.025437 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:34.025355    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6947f4bd9f-px842: secret "image-registry-tls" not found
Jul 18 16:40:34.025437 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:34.025417    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls podName:26cc0844-d905-444b-bea5-d6b4a2b148a2 nodeName:}" failed. No retries permitted until 2026-07-18 16:40:36.025400192 +0000 UTC m=+254.723230872 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls") pod "image-registry-6947f4bd9f-px842" (UID: "26cc0844-d905-444b-bea5-d6b4a2b148a2") : secret "image-registry-tls" not found
Jul 18 16:40:34.510965 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:34.510923    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:35.510632 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.510592    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kvsvt" event={"ID":"05af46b3-3c72-416b-b837-d9412910c1b5","Type":"ContainerStarted","Data":"a770cdfde263c11cfc6154dca7dcb9ea2cfbdf8b70efedf58167652416887444"}
Jul 18 16:40:35.528212 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.528161    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-kvsvt" podStartSLOduration=1.431867861 podStartE2EDuration="3.528142357s" totalImagesPullingTime="2.096274496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:40:35.52705648 +0000 UTC m=+254.224887164" watchObservedRunningTime="2026-07-18 16:40:35.528142357 +0000 UTC m=+254.225973042"
Jul 18 16:40:35.637180 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.637148    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w95dc"]
Jul 18 16:40:35.640275 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.640254    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:35.643104 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.643082    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 18 16:40:35.643192 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.643139    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k6m7m\""
Jul 18 16:40:35.643192 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.643147    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 18 16:40:35.651218 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.651198    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w95dc"]
Jul 18 16:40:35.664196 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.664153    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:35.740562 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.740531    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:35.740562 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.740564    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpw8g\" (UniqueName: \"kubernetes.io/projected/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-api-access-jpw8g\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:35.740752 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.740594    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0728aabe-3077-4396-98c2-f6f7c1ec84db-crio-socket\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:35.740752 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.740692    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:35.740752 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.740734    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0728aabe-3077-4396-98c2-f6f7c1ec84db-data-volume\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:35.841385 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.841358    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:35.841551 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.841405    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0728aabe-3077-4396-98c2-f6f7c1ec84db-data-volume\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:35.841551 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.841452    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:35.841551 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.841475    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpw8g\" (UniqueName: \"kubernetes.io/projected/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-api-access-jpw8g\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:35.841551 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.841501    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0728aabe-3077-4396-98c2-f6f7c1ec84db-crio-socket\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:35.841551 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:35.841529    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 18 16:40:35.841738 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.841569    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0728aabe-3077-4396-98c2-f6f7c1ec84db-crio-socket\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:35.841738 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:35.841576    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 18 16:40:35.841738 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:35.841610    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls podName:0728aabe-3077-4396-98c2-f6f7c1ec84db nodeName:}" failed. No retries permitted until 2026-07-18 16:40:36.341589164 +0000 UTC m=+255.039419838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w95dc" (UID: "0728aabe-3077-4396-98c2-f6f7c1ec84db") : secret "insights-runtime-extractor-tls" not found
Jul 18 16:40:35.841738 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:35.841651    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm podName:0728aabe-3077-4396-98c2-f6f7c1ec84db nodeName:}" failed. No retries permitted until 2026-07-18 16:40:36.341632684 +0000 UTC m=+255.039463370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w95dc" (UID: "0728aabe-3077-4396-98c2-f6f7c1ec84db") : configmap "kube-rbac-proxy" not found
Jul 18 16:40:35.841924 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.841772    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0728aabe-3077-4396-98c2-f6f7c1ec84db-data-volume\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:35.850132 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:35.850113    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpw8g\" (UniqueName: \"kubernetes.io/projected/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-api-access-jpw8g\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:36.043218 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:36.043187    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:36.043386 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:36.043297    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 18 16:40:36.043386 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:36.043309    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6947f4bd9f-px842: secret "image-registry-tls" not found
Jul 18 16:40:36.043386 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:36.043366    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls podName:26cc0844-d905-444b-bea5-d6b4a2b148a2 nodeName:}" failed. No retries permitted until 2026-07-18 16:40:40.043351683 +0000 UTC m=+258.741182345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls") pod "image-registry-6947f4bd9f-px842" (UID: "26cc0844-d905-444b-bea5-d6b4a2b148a2") : secret "image-registry-tls" not found
Jul 18 16:40:36.345675 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:36.345632    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:36.345865 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:36.345728    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:36.345865 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:36.345782    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 18 16:40:36.345865 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:36.345811    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 18 16:40:36.345865 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:36.345859    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls podName:0728aabe-3077-4396-98c2-f6f7c1ec84db nodeName:}" failed. No retries permitted until 2026-07-18 16:40:37.345829786 +0000 UTC m=+256.043660448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w95dc" (UID: "0728aabe-3077-4396-98c2-f6f7c1ec84db") : secret "insights-runtime-extractor-tls" not found
Jul 18 16:40:36.346045 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:36.345881    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm podName:0728aabe-3077-4396-98c2-f6f7c1ec84db nodeName:}" failed. No retries permitted until 2026-07-18 16:40:37.345866084 +0000 UTC m=+256.043696760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w95dc" (UID: "0728aabe-3077-4396-98c2-f6f7c1ec84db") : configmap "kube-rbac-proxy" not found
Jul 18 16:40:36.890196 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:36.890166    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:36.990948 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:36.990920    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:37.352370 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.352339    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:37.352527 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.352427    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:37.352527 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:37.352496    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 18 16:40:37.352527 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:37.352505    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 18 16:40:37.352624 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:37.352563    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls podName:0728aabe-3077-4396-98c2-f6f7c1ec84db nodeName:}" failed. No retries permitted until 2026-07-18 16:40:39.352545078 +0000 UTC m=+258.050375743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w95dc" (UID: "0728aabe-3077-4396-98c2-f6f7c1ec84db") : secret "insights-runtime-extractor-tls" not found
Jul 18 16:40:37.352624 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:37.352580    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm podName:0728aabe-3077-4396-98c2-f6f7c1ec84db nodeName:}" failed. No retries permitted until 2026-07-18 16:40:39.352572471 +0000 UTC m=+258.050403140 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w95dc" (UID: "0728aabe-3077-4396-98c2-f6f7c1ec84db") : configmap "kube-rbac-proxy" not found
Jul 18 16:40:37.611898 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.611813    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sz6hk_8c71dd87-f690-4c8e-b286-6eff3854fd93/dns-node-resolver/0.log"
Jul 18 16:40:37.615150 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.615124    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:37.625101 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.625080    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-ml29g"]
Jul 18 16:40:37.628116 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.628101    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-ml29g"
Jul 18 16:40:37.630824 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.630805    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 18 16:40:37.630898 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.630839    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4g8bg\""
Jul 18 16:40:37.632033 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.632011    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 18 16:40:37.637080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.637061    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-ml29g"]
Jul 18 16:40:37.713053 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.713007    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:37.756158 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.756135    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2whl\" (UniqueName: \"kubernetes.io/projected/5be0e73f-85a3-44af-b6fa-3a385abaf6c4-kube-api-access-h2whl\") pod \"migrator-7cc9c7ffd7-ml29g\" (UID: \"5be0e73f-85a3-44af-b6fa-3a385abaf6c4\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-ml29g"
Jul 18 16:40:37.808430 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.808394    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:37.856928 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.856898    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2whl\" (UniqueName: \"kubernetes.io/projected/5be0e73f-85a3-44af-b6fa-3a385abaf6c4-kube-api-access-h2whl\") pod \"migrator-7cc9c7ffd7-ml29g\" (UID: \"5be0e73f-85a3-44af-b6fa-3a385abaf6c4\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-ml29g"
Jul 18 16:40:37.864934 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.864871    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2whl\" (UniqueName: \"kubernetes.io/projected/5be0e73f-85a3-44af-b6fa-3a385abaf6c4-kube-api-access-h2whl\") pod \"migrator-7cc9c7ffd7-ml29g\" (UID: \"5be0e73f-85a3-44af-b6fa-3a385abaf6c4\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-ml29g"
Jul 18 16:40:37.936959 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:37.936923    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-ml29g"
Jul 18 16:40:38.058800 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:38.058704    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-ml29g"]
Jul 18 16:40:38.061125 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:40:38.061099    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be0e73f_85a3_44af_b6fa_3a385abaf6c4.slice/crio-d180cbf3f8c9581e287c854b2dac5b6b6f0733c4c9acb6fedce8ab4570c00fb6 WatchSource:0}: Error finding container d180cbf3f8c9581e287c854b2dac5b6b6f0733c4c9acb6fedce8ab4570c00fb6: Status 404 returned error can't find the container with id d180cbf3f8c9581e287c854b2dac5b6b6f0733c4c9acb6fedce8ab4570c00fb6
Jul 18 16:40:38.520506 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:38.520472    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-ml29g" event={"ID":"5be0e73f-85a3-44af-b6fa-3a385abaf6c4","Type":"ContainerStarted","Data":"d180cbf3f8c9581e287c854b2dac5b6b6f0733c4c9acb6fedce8ab4570c00fb6"}
Jul 18 16:40:38.604035 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:38.604009    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6jtwc_a48a86ca-3a85-46b8-ada3-0125ec0e9546/node-ca/0.log"
Jul 18 16:40:39.235820 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:39.235748    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:39.370431 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:39.370402    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:39.370557 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:39.370454    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:39.370557 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:39.370539    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 18 16:40:39.370666 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:39.370557    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 18 16:40:39.370666 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:39.370602    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls podName:0728aabe-3077-4396-98c2-f6f7c1ec84db nodeName:}" failed. No retries permitted until 2026-07-18 16:40:43.370588683 +0000 UTC m=+262.068419345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w95dc" (UID: "0728aabe-3077-4396-98c2-f6f7c1ec84db") : secret "insights-runtime-extractor-tls" not found
Jul 18 16:40:39.370666 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:39.370617    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm podName:0728aabe-3077-4396-98c2-f6f7c1ec84db nodeName:}" failed. No retries permitted until 2026-07-18 16:40:43.370608325 +0000 UTC m=+262.068438986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w95dc" (UID: "0728aabe-3077-4396-98c2-f6f7c1ec84db") : configmap "kube-rbac-proxy" not found
Jul 18 16:40:39.524976 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:39.524890    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-ml29g" event={"ID":"5be0e73f-85a3-44af-b6fa-3a385abaf6c4","Type":"ContainerStarted","Data":"5a01d0b3e71931fa4e28e686ef9d4768697d261efd660553a26888bd04d845c1"}
Jul 18 16:40:39.524976 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:39.524950    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-ml29g" event={"ID":"5be0e73f-85a3-44af-b6fa-3a385abaf6c4","Type":"ContainerStarted","Data":"48e8956591548ffdd5b194b74a1cf4d58cc147024f94cd9abf8ee302c5799845"}
Jul 18 16:40:39.539792 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:39.539752    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-ml29g" podStartSLOduration=1.377502706 podStartE2EDuration="2.539737939s" totalImagesPullingTime="1.162235233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:40:39.539252685 +0000 UTC m=+258.237083370" watchObservedRunningTime="2026-07-18 16:40:39.539737939 +0000 UTC m=+258.237568675"
Jul 18 16:40:39.604388 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:39.604361    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-psptl_09878619-4f85-4acb-bce3-94a409e582d1/serve-healthcheck-canary/0.log"
Jul 18 16:40:39.606490 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:39.606463    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:39.708985 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:39.708940    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:40.076855 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:40.076821    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:40.077011 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:40.076974    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 18 16:40:40.077011 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:40.076989    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6947f4bd9f-px842: secret "image-registry-tls" not found
Jul 18 16:40:40.077084 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:40.077041    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls podName:26cc0844-d905-444b-bea5-d6b4a2b148a2 nodeName:}" failed. No retries permitted until 2026-07-18 16:40:48.077026306 +0000 UTC m=+266.774856968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls") pod "image-registry-6947f4bd9f-px842" (UID: "26cc0844-d905-444b-bea5-d6b4a2b148a2") : secret "image-registry-tls" not found
Jul 18 16:40:40.119300 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:40.119268    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:42.831506 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:42.831407    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:42.939366 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:42.939332    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:43.039957 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:43.039925    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:43.143781 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:43.143710    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:43.240572 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:43.240539    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:43.340472 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:43.340442    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:43.403987 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:43.403916    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:43.403987 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:43.403965    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:43.404156 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:43.404025    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 18 16:40:43.404156 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:43.404042    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 18 16:40:43.404156 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:43.404085    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm podName:0728aabe-3077-4396-98c2-f6f7c1ec84db nodeName:}" failed. No retries permitted until 2026-07-18 16:40:51.404066129 +0000 UTC m=+270.101896794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w95dc" (UID: "0728aabe-3077-4396-98c2-f6f7c1ec84db") : configmap "kube-rbac-proxy" not found
Jul 18 16:40:43.404156 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:43.404111    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls podName:0728aabe-3077-4396-98c2-f6f7c1ec84db nodeName:}" failed. No retries permitted until 2026-07-18 16:40:51.404101706 +0000 UTC m=+270.101932374 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w95dc" (UID: "0728aabe-3077-4396-98c2-f6f7c1ec84db") : secret "insights-runtime-extractor-tls" not found
Jul 18 16:40:43.925319 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:43.925291    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:44.123379 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:44.123342    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:44.718031 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:44.718000    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:45.123334 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:45.123307    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:45.525823 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:45.525758    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:45.728035 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:45.728005    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:46.123950 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:46.123918    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:46.589563 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:46.589534    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:46.708116 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:46.708088    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:46.768538 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:46.768508    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:46.922897 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:46.922797    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:47.324547 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:47.324480    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:47.523772 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:47.523741    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:47.719273 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:47.719231    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:47.927636 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:47.927597    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:48.143401 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:48.143364    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:48.145806 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:48.145781    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls\") pod \"image-registry-6947f4bd9f-px842\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") " pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:48.196857 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:48.196822    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:48.324419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:48.324396    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6947f4bd9f-px842"]
Jul 18 16:40:48.326793 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:40:48.326771    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cc0844_d905_444b_bea5_d6b4a2b148a2.slice/crio-79f8309be19a257b7c6ba924ffabad657ce7a65273c3d55c5d18cb11795a7ccc WatchSource:0}: Error finding container 79f8309be19a257b7c6ba924ffabad657ce7a65273c3d55c5d18cb11795a7ccc: Status 404 returned error can't find the container with id 79f8309be19a257b7c6ba924ffabad657ce7a65273c3d55c5d18cb11795a7ccc
Jul 18 16:40:48.551813 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:48.551730    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6947f4bd9f-px842" event={"ID":"26cc0844-d905-444b-bea5-d6b4a2b148a2","Type":"ContainerStarted","Data":"89384b111304be2d66221da49d02c511244377592900a15f994acd5aabddbd5e"}
Jul 18 16:40:48.551813 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:48.551779    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6947f4bd9f-px842" event={"ID":"26cc0844-d905-444b-bea5-d6b4a2b148a2","Type":"ContainerStarted","Data":"79f8309be19a257b7c6ba924ffabad657ce7a65273c3d55c5d18cb11795a7ccc"}
Jul 18 16:40:48.551813 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:48.551810    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:40:48.570752 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:48.570706    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6947f4bd9f-px842" podStartSLOduration=16.57069135 podStartE2EDuration="16.57069135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:40:48.56998737 +0000 UTC m=+267.267818065" watchObservedRunningTime="2026-07-18 16:40:48.57069135 +0000 UTC m=+267.268522037"
Jul 18 16:40:48.716296 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:48.716262    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:48.915038 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:48.915006    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:48.965523 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:48.965493    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:49.316433 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:49.316361    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:49.522920 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:49.522888    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:50.129137 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:50.129102    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:50.407887 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:50.407799    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:50.525736 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:50.525708    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:50.720005 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:50.719932    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:51.470425 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:51.470393    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:51.470605 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:51.470451    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:51.470605 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:51.470558    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 18 16:40:51.470680 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:40:51.470618    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls podName:0728aabe-3077-4396-98c2-f6f7c1ec84db nodeName:}" failed. No retries permitted until 2026-07-18 16:41:07.470603669 +0000 UTC m=+286.168434331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w95dc" (UID: "0728aabe-3077-4396-98c2-f6f7c1ec84db") : secret "insights-runtime-extractor-tls" not found
Jul 18 16:40:51.470944 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:51.470923    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0728aabe-3077-4396-98c2-f6f7c1ec84db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:40:51.518945 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:51.518921    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:52.121359 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:52.121330    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:52.320176 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:52.320146    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:52.518807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:52.518732    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:52.722295 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:52.722262    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:53.323164 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:53.323129    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:53.520710 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:53.520675    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:57.729820 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.729787    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6947f4bd9f-px842"]
Jul 18 16:40:57.776755 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.776727    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"]
Jul 18 16:40:57.781392 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.781374    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.800776 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.800751    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"]
Jul 18 16:40:57.815933 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.815905    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95484bc2-9453-4837-8f68-45a375d918bf-bound-sa-token\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.816028 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.815944    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95484bc2-9453-4837-8f68-45a375d918bf-trusted-ca\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.816028 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.815970    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95484bc2-9453-4837-8f68-45a375d918bf-installation-pull-secrets\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.816116 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.816048    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95484bc2-9453-4837-8f68-45a375d918bf-image-registry-private-configuration\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.816116 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.816082    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95484bc2-9453-4837-8f68-45a375d918bf-registry-tls\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.816116 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.816110    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vts5w\" (UniqueName: \"kubernetes.io/projected/95484bc2-9453-4837-8f68-45a375d918bf-kube-api-access-vts5w\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.816232 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.816129    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95484bc2-9453-4837-8f68-45a375d918bf-registry-certificates\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.816232 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.816153    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95484bc2-9453-4837-8f68-45a375d918bf-ca-trust-extracted\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.850036 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.850011    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:57.917426 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.917396    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95484bc2-9453-4837-8f68-45a375d918bf-bound-sa-token\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.917426 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.917427    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95484bc2-9453-4837-8f68-45a375d918bf-trusted-ca\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.917601 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.917444    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95484bc2-9453-4837-8f68-45a375d918bf-installation-pull-secrets\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.917601 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.917466    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95484bc2-9453-4837-8f68-45a375d918bf-image-registry-private-configuration\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.917601 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.917511    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95484bc2-9453-4837-8f68-45a375d918bf-registry-tls\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.917601 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.917549    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vts5w\" (UniqueName: \"kubernetes.io/projected/95484bc2-9453-4837-8f68-45a375d918bf-kube-api-access-vts5w\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.917601 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.917579    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95484bc2-9453-4837-8f68-45a375d918bf-registry-certificates\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.917885 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.917614    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95484bc2-9453-4837-8f68-45a375d918bf-ca-trust-extracted\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.918104 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.918082    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/95484bc2-9453-4837-8f68-45a375d918bf-ca-trust-extracted\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.918496 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.918472    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/95484bc2-9453-4837-8f68-45a375d918bf-trusted-ca\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.918573 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.918472    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/95484bc2-9453-4837-8f68-45a375d918bf-registry-certificates\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.920082 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.920060    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/95484bc2-9453-4837-8f68-45a375d918bf-image-registry-private-configuration\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.920195 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.920127    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/95484bc2-9453-4837-8f68-45a375d918bf-installation-pull-secrets\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.920256 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.920232    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/95484bc2-9453-4837-8f68-45a375d918bf-registry-tls\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.925793 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.925774    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/95484bc2-9453-4837-8f68-45a375d918bf-bound-sa-token\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.925912 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.925893    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vts5w\" (UniqueName: \"kubernetes.io/projected/95484bc2-9453-4837-8f68-45a375d918bf-kube-api-access-vts5w\") pod \"image-registry-7dcbf79b9c-x7qxx\" (UID: \"95484bc2-9453-4837-8f68-45a375d918bf\") " pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:57.949819 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:57.949794    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:40:58.089192 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:58.089169    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:58.215289 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:58.215262    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"]
Jul 18 16:40:58.218743 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:40:58.218717    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95484bc2_9453_4837_8f68_45a375d918bf.slice/crio-24dec6ad18ac2039379364751cccb59fa1e51fdf3928ed6c62e1cd01bf29140a WatchSource:0}: Error finding container 24dec6ad18ac2039379364751cccb59fa1e51fdf3928ed6c62e1cd01bf29140a: Status 404 returned error can't find the container with id 24dec6ad18ac2039379364751cccb59fa1e51fdf3928ed6c62e1cd01bf29140a
Jul 18 16:40:58.583165 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:58.583129    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx" event={"ID":"95484bc2-9453-4837-8f68-45a375d918bf","Type":"ContainerStarted","Data":"ddf7d7cb628d8a887992c9153763d07907da9552994fe75b45167aedcd6f597d"}
Jul 18 16:40:58.583339 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:58.583187    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx" event={"ID":"95484bc2-9453-4837-8f68-45a375d918bf","Type":"ContainerStarted","Data":"24dec6ad18ac2039379364751cccb59fa1e51fdf3928ed6c62e1cd01bf29140a"}
Jul 18 16:40:58.583339 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:58.583216    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:40:58.601343 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:40:58.601297    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx" podStartSLOduration=1.601281094 podStartE2EDuration="1.601281094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:40:58.60039256 +0000 UTC m=+277.298223245" watchObservedRunningTime="2026-07-18 16:40:58.601281094 +0000 UTC m=+277.299111777"
Jul 18 16:41:00.498177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:00.498141    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:00.554893 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:00.554869    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:03.272276 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:41:03.272232    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-994h4" podUID="9bbd0e94-6c81-4eaa-abb9-522600419f63"
Jul 18 16:41:03.598511 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:03.598483    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-994h4"
Jul 18 16:41:06.778163 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:06.778131    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:06.878054 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:06.878025    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:06.927999 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:06.927966    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xmhf6"]
Jul 18 16:41:06.932461 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:06.932446    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xmhf6"
Jul 18 16:41:06.935062 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:06.935042    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5rmpf\""
Jul 18 16:41:06.935189 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:06.935081    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 18 16:41:06.939302 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:06.939172    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xmhf6"]
Jul 18 16:41:06.941950 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:06.941930    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:06.989948 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:06.989921    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9bdcba4c-e0df-485e-a4be-05d82d176e0a-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-xmhf6\" (UID: \"9bdcba4c-e0df-485e-a4be-05d82d176e0a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xmhf6"
Jul 18 16:41:07.090925 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.090892    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9bdcba4c-e0df-485e-a4be-05d82d176e0a-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-xmhf6\" (UID: \"9bdcba4c-e0df-485e-a4be-05d82d176e0a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xmhf6"
Jul 18 16:41:07.091067 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:41:07.091046    2554 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 18 16:41:07.091122 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:41:07.091114    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bdcba4c-e0df-485e-a4be-05d82d176e0a-tls-certificates podName:9bdcba4c-e0df-485e-a4be-05d82d176e0a nodeName:}" failed. No retries permitted until 2026-07-18 16:41:07.591097749 +0000 UTC m=+286.288928410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/9bdcba4c-e0df-485e-a4be-05d82d176e0a-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-xmhf6" (UID: "9bdcba4c-e0df-485e-a4be-05d82d176e0a") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 18 16:41:07.191419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.191387    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:41:07.193836 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.193811    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bbd0e94-6c81-4eaa-abb9-522600419f63-metrics-tls\") pod \"dns-default-994h4\" (UID: \"9bbd0e94-6c81-4eaa-abb9-522600419f63\") " pod="openshift-dns/dns-default-994h4"
Jul 18 16:41:07.202400 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.202382    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2jrkz\""
Jul 18 16:41:07.210314 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.210302    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-994h4"
Jul 18 16:41:07.332536 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.332513    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-994h4"]
Jul 18 16:41:07.334709 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:41:07.334678    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbd0e94_6c81_4eaa_abb9_522600419f63.slice/crio-fa2b35db153ec9d85f29303709f135098554d3cc4d32ba191db51680601ce730 WatchSource:0}: Error finding container fa2b35db153ec9d85f29303709f135098554d3cc4d32ba191db51680601ce730: Status 404 returned error can't find the container with id fa2b35db153ec9d85f29303709f135098554d3cc4d32ba191db51680601ce730
Jul 18 16:41:07.366142 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.366067    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:07.493824 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.493793    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:41:07.496150 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.496133    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0728aabe-3077-4396-98c2-f6f7c1ec84db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w95dc\" (UID: \"0728aabe-3077-4396-98c2-f6f7c1ec84db\") " pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:41:07.594119 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.594088    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9bdcba4c-e0df-485e-a4be-05d82d176e0a-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-xmhf6\" (UID: \"9bdcba4c-e0df-485e-a4be-05d82d176e0a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xmhf6"
Jul 18 16:41:07.596481 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.596464    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9bdcba4c-e0df-485e-a4be-05d82d176e0a-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-xmhf6\" (UID: \"9bdcba4c-e0df-485e-a4be-05d82d176e0a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xmhf6"
Jul 18 16:41:07.610635 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.610613    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-994h4" event={"ID":"9bbd0e94-6c81-4eaa-abb9-522600419f63","Type":"ContainerStarted","Data":"fa2b35db153ec9d85f29303709f135098554d3cc4d32ba191db51680601ce730"}
Jul 18 16:41:07.736704 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.736630    2554 patch_prober.go:28] interesting pod/image-registry-6947f4bd9f-px842 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 18 16:41:07.736704 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.736683    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6947f4bd9f-px842" podUID="26cc0844-d905-444b-bea5-d6b4a2b148a2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 18 16:41:07.749351 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.749322    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w95dc"
Jul 18 16:41:07.842524 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.842231    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xmhf6"
Jul 18 16:41:07.895658 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.895626    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w95dc"]
Jul 18 16:41:07.898533 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:41:07.898492    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0728aabe_3077_4396_98c2_f6f7c1ec84db.slice/crio-62f2cfcab56364610a32852e1db7249e445065392e4f449e4ba1354c477d7c33 WatchSource:0}: Error finding container 62f2cfcab56364610a32852e1db7249e445065392e4f449e4ba1354c477d7c33: Status 404 returned error can't find the container with id 62f2cfcab56364610a32852e1db7249e445065392e4f449e4ba1354c477d7c33
Jul 18 16:41:07.987688 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:07.987612    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xmhf6"]
Jul 18 16:41:07.991561 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:41:07.991536    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdcba4c_e0df_485e_a4be_05d82d176e0a.slice/crio-b56c5f78ae10ec2e695d890e03492d1515781684a1482c46b47f53a397a4956d WatchSource:0}: Error finding container b56c5f78ae10ec2e695d890e03492d1515781684a1482c46b47f53a397a4956d: Status 404 returned error can't find the container with id b56c5f78ae10ec2e695d890e03492d1515781684a1482c46b47f53a397a4956d
Jul 18 16:41:08.618412 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:08.618338    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xmhf6" event={"ID":"9bdcba4c-e0df-485e-a4be-05d82d176e0a","Type":"ContainerStarted","Data":"b56c5f78ae10ec2e695d890e03492d1515781684a1482c46b47f53a397a4956d"}
Jul 18 16:41:08.620073 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:08.620044    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w95dc" event={"ID":"0728aabe-3077-4396-98c2-f6f7c1ec84db","Type":"ContainerStarted","Data":"62f2cfcab56364610a32852e1db7249e445065392e4f449e4ba1354c477d7c33"}
Jul 18 16:41:08.672224 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:08.672151    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:08.772323 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:08.772241    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:09.632010 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:09.631975    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-994h4" event={"ID":"9bbd0e94-6c81-4eaa-abb9-522600419f63","Type":"ContainerStarted","Data":"959da8f814dab5529d8bff4b7ae70a2540f810285add9636a37d9b51ee25ab22"}
Jul 18 16:41:09.632472 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:09.632127    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-994h4" event={"ID":"9bbd0e94-6c81-4eaa-abb9-522600419f63","Type":"ContainerStarted","Data":"ad0ec297e3184230dd5f75ecfe2d53c6886df76287a70ea809872c5e64557af2"}
Jul 18 16:41:09.632472 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:09.632339    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-994h4"
Jul 18 16:41:09.649129 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:09.648864    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-994h4" podStartSLOduration=250.893898277 podStartE2EDuration="4m12.648824859s" totalImagesPullingTime="1.754926582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:36:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:41:09.647774608 +0000 UTC m=+288.345605292" watchObservedRunningTime="2026-07-18 16:41:09.648824859 +0000 UTC m=+288.346655547"
Jul 18 16:41:10.639132 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:10.639096    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xmhf6" event={"ID":"9bdcba4c-e0df-485e-a4be-05d82d176e0a","Type":"ContainerStarted","Data":"7ffb7e94f9abfebebad1186fc8facd2e45f32796e36dd678172214035e5d4d2d"}
Jul 18 16:41:10.639607 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:10.639183    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xmhf6"
Jul 18 16:41:10.644200 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:10.644180    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xmhf6"
Jul 18 16:41:10.654730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:10.654501    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-xmhf6" podStartSLOduration=2.767827613 podStartE2EDuration="4.654482317s" totalImagesPullingTime="1.886654704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:41:10.654206409 +0000 UTC m=+289.352037095" watchObservedRunningTime="2026-07-18 16:41:10.654482317 +0000 UTC m=+289.352313002"
Jul 18 16:41:10.951308 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:10.951234    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:10.982376 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:10.982347    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"]
Jul 18 16:41:10.999923 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:10.999891    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"]
Jul 18 16:41:11.000081 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:10.999970    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:11.002480 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.002456    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 18 16:41:11.002710 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.002688    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 18 16:41:11.002834 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.002720    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 18 16:41:11.002834 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.002721    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-v9spz\""
Jul 18 16:41:11.002834 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.002741    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 18 16:41:11.003921 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.003901    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 18 16:41:11.021835 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.021801    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:11.126221 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.126187    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdhch\" (UniqueName: \"kubernetes.io/projected/ea4f34b8-1180-4804-923e-c5868ea12113-kube-api-access-wdhch\") pod \"prometheus-operator-5945b5cfd-8pklg\" (UID: \"ea4f34b8-1180-4804-923e-c5868ea12113\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:11.126400 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.126245    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea4f34b8-1180-4804-923e-c5868ea12113-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8pklg\" (UID: \"ea4f34b8-1180-4804-923e-c5868ea12113\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:11.126400 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.126288    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea4f34b8-1180-4804-923e-c5868ea12113-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8pklg\" (UID: \"ea4f34b8-1180-4804-923e-c5868ea12113\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:11.126400 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.126358    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea4f34b8-1180-4804-923e-c5868ea12113-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8pklg\" (UID: \"ea4f34b8-1180-4804-923e-c5868ea12113\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:11.227740 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.227655    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea4f34b8-1180-4804-923e-c5868ea12113-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8pklg\" (UID: \"ea4f34b8-1180-4804-923e-c5868ea12113\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:11.227952 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.227750    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea4f34b8-1180-4804-923e-c5868ea12113-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8pklg\" (UID: \"ea4f34b8-1180-4804-923e-c5868ea12113\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:11.227952 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.227789    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdhch\" (UniqueName: \"kubernetes.io/projected/ea4f34b8-1180-4804-923e-c5868ea12113-kube-api-access-wdhch\") pod \"prometheus-operator-5945b5cfd-8pklg\" (UID: \"ea4f34b8-1180-4804-923e-c5868ea12113\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:11.227952 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.227824    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea4f34b8-1180-4804-923e-c5868ea12113-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8pklg\" (UID: \"ea4f34b8-1180-4804-923e-c5868ea12113\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:11.227952 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:41:11.227937    2554 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 18 16:41:11.228140 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:41:11.227999    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea4f34b8-1180-4804-923e-c5868ea12113-prometheus-operator-tls podName:ea4f34b8-1180-4804-923e-c5868ea12113 nodeName:}" failed. No retries permitted until 2026-07-18 16:41:11.72798096 +0000 UTC m=+290.425811623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/ea4f34b8-1180-4804-923e-c5868ea12113-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-8pklg" (UID: "ea4f34b8-1180-4804-923e-c5868ea12113") : secret "prometheus-operator-tls" not found
Jul 18 16:41:11.228423 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.228401    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea4f34b8-1180-4804-923e-c5868ea12113-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8pklg\" (UID: \"ea4f34b8-1180-4804-923e-c5868ea12113\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:11.230916 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.230884    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea4f34b8-1180-4804-923e-c5868ea12113-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8pklg\" (UID: \"ea4f34b8-1180-4804-923e-c5868ea12113\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:11.237132 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.237086    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdhch\" (UniqueName: \"kubernetes.io/projected/ea4f34b8-1180-4804-923e-c5868ea12113-kube-api-access-wdhch\") pod \"prometheus-operator-5945b5cfd-8pklg\" (UID: \"ea4f34b8-1180-4804-923e-c5868ea12113\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:11.733993 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.733954    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea4f34b8-1180-4804-923e-c5868ea12113-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8pklg\" (UID: \"ea4f34b8-1180-4804-923e-c5868ea12113\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:11.734453 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:41:11.734135    2554 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 18 16:41:11.734453 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:41:11.734215    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea4f34b8-1180-4804-923e-c5868ea12113-prometheus-operator-tls podName:ea4f34b8-1180-4804-923e-c5868ea12113 nodeName:}" failed. No retries permitted until 2026-07-18 16:41:12.734194046 +0000 UTC m=+291.432024714 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/ea4f34b8-1180-4804-923e-c5868ea12113-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-8pklg" (UID: "ea4f34b8-1180-4804-923e-c5868ea12113") : secret "prometheus-operator-tls" not found
Jul 18 16:41:11.812164 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:11.812127    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:12.743662 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:12.743625    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea4f34b8-1180-4804-923e-c5868ea12113-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8pklg\" (UID: \"ea4f34b8-1180-4804-923e-c5868ea12113\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:12.746452 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:12.746425    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea4f34b8-1180-4804-923e-c5868ea12113-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8pklg\" (UID: \"ea4f34b8-1180-4804-923e-c5868ea12113\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:12.810701 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:12.810658    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"
Jul 18 16:41:13.049159 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:13.049135    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8pklg"]
Jul 18 16:41:13.053150 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:41:13.053120    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4f34b8_1180_4804_923e_c5868ea12113.slice/crio-a76e412db7fd68c980d4acc4b03a37268372b5ff800b9ff933d069008c829b2a WatchSource:0}: Error finding container a76e412db7fd68c980d4acc4b03a37268372b5ff800b9ff933d069008c829b2a: Status 404 returned error can't find the container with id a76e412db7fd68c980d4acc4b03a37268372b5ff800b9ff933d069008c829b2a
Jul 18 16:41:13.651465 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:13.651426    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg" event={"ID":"ea4f34b8-1180-4804-923e-c5868ea12113","Type":"ContainerStarted","Data":"a76e412db7fd68c980d4acc4b03a37268372b5ff800b9ff933d069008c829b2a"}
Jul 18 16:41:13.653498 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:13.653454    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w95dc" event={"ID":"0728aabe-3077-4396-98c2-f6f7c1ec84db","Type":"ContainerStarted","Data":"0d14fe3550c059fe2e721392a41b91bf09de2728391b09b2efe4361ec55af378"}
Jul 18 16:41:15.760581 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:41:15.760544    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hczmz" podUID="2f9ff934-e093-4adb-b818-279e624555de"
Jul 18 16:41:16.664960 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:16.664869    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg" event={"ID":"ea4f34b8-1180-4804-923e-c5868ea12113","Type":"ContainerStarted","Data":"93193e68322172cd11728d5a1edf1ab82992b0c43b2260f75597ab59a6d95406"}
Jul 18 16:41:16.664960 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:16.664939    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg" event={"ID":"ea4f34b8-1180-4804-923e-c5868ea12113","Type":"ContainerStarted","Data":"2f239227ea814597cc5e560bfa2f6f827186c643bbe8600b9257753a2f8e96e0"}
Jul 18 16:41:16.666661 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:16.666636    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w95dc" event={"ID":"0728aabe-3077-4396-98c2-f6f7c1ec84db","Type":"ContainerStarted","Data":"6ffceca1022e9674b3d7cc11081dcc69c0d4cab5a7adcad401cdce6118803716"}
Jul 18 16:41:16.666946 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:16.666927    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:16.680650 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:16.680603    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8pklg" podStartSLOduration=3.327021739 podStartE2EDuration="6.680591482s" totalImagesPullingTime="3.353569743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:41:16.679301683 +0000 UTC m=+295.377132367" watchObservedRunningTime="2026-07-18 16:41:16.680591482 +0000 UTC m=+295.378422143"
Jul 18 16:41:17.735412 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:17.735379    2554 patch_prober.go:28] interesting pod/image-registry-6947f4bd9f-px842 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 18 16:41:17.735958 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:17.735431    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6947f4bd9f-px842" podUID="26cc0844-d905-444b-bea5-d6b4a2b148a2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 18 16:41:18.084666 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.084639    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:18.093822 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.093793    2554 patch_prober.go:28] interesting pod/image-registry-7dcbf79b9c-x7qxx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 18 16:41:18.094032 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.093868    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx" podUID="95484bc2-9453-4837-8f68-45a375d918bf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 18 16:41:18.285616 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.285561    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:18.347820 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.347745    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mwgzb"]
Jul 18 16:41:18.352575 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.352546    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.355157 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.355133    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 18 16:41:18.355157 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.355148    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8ss7d\""
Jul 18 16:41:18.355357 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.355179    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 18 16:41:18.355357 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.355153    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 18 16:41:18.375082 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.375057    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:18.451343 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.451297    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:18.494932 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.494900    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-wtmp\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.495115 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.494936    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-tls\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.495115 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.494967    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.495115 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.495054    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-root\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.495115 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.495109    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-accelerators-collector-config\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.495306 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.495139    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-metrics-client-ca\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.495306 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.495169    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm2tv\" (UniqueName: \"kubernetes.io/projected/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-kube-api-access-hm2tv\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.495306 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.495194    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-sys\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.495306 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.495259    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-textfile\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.553814 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.553773    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:18.595969 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.595938    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-sys\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.596148 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.595990    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-textfile\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.596148 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.596030    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-wtmp\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.596148 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.596053    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-tls\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.596148 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.596057    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-sys\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.596148 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.596075    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.596148 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.596129    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-root\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.596454 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:41:18.596171    2554 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 18 16:41:18.596454 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.596179    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-accelerators-collector-config\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.596454 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.596209    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-wtmp\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.596454 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.596209    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-metrics-client-ca\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.596454 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:41:18.596232    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-tls podName:57df3c7d-e80d-49b0-b1d0-4d21cbfc519f nodeName:}" failed. No retries permitted until 2026-07-18 16:41:19.096212942 +0000 UTC m=+297.794043605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-tls") pod "node-exporter-mwgzb" (UID: "57df3c7d-e80d-49b0-b1d0-4d21cbfc519f") : secret "node-exporter-tls" not found
Jul 18 16:41:18.596454 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.596251    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-root\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.596454 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.596350    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm2tv\" (UniqueName: \"kubernetes.io/projected/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-kube-api-access-hm2tv\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.596454 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.596391    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-textfile\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.596815 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.596795    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-metrics-client-ca\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.596815 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.596808    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-accelerators-collector-config\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.598678 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.598626    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.605083 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.605063    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm2tv\" (UniqueName: \"kubernetes.io/projected/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-kube-api-access-hm2tv\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:18.652370 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.652325    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:18.816612 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:18.816580    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:19.023464 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:19.023375    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:19.100573 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:19.100542    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-tls\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:19.103017 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:19.102995    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/57df3c7d-e80d-49b0-b1d0-4d21cbfc519f-node-exporter-tls\") pod \"node-exporter-mwgzb\" (UID: \"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f\") " pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:19.263430 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:19.263399    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mwgzb"
Jul 18 16:41:19.303971 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:19.303942    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:19.384985 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:19.384937    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:19.504502 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:19.504458    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:19.592800 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:19.592763    2554 patch_prober.go:28] interesting pod/image-registry-7dcbf79b9c-x7qxx container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 18 16:41:19.592989 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:19.592828    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx" podUID="95484bc2-9453-4837-8f68-45a375d918bf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 18 16:41:19.641816 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:19.641790    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-994h4"
Jul 18 16:41:19.941942 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:41:19.941912    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57df3c7d_e80d_49b0_b1d0_4d21cbfc519f.slice/crio-8cea8501c6d3d21df7368a9b478e32e9f1d9875c2f5ebd237671d757531da604 WatchSource:0}: Error finding container 8cea8501c6d3d21df7368a9b478e32e9f1d9875c2f5ebd237671d757531da604: Status 404 returned error can't find the container with id 8cea8501c6d3d21df7368a9b478e32e9f1d9875c2f5ebd237671d757531da604
Jul 18 16:41:20.685357 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:20.685324    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mwgzb" event={"ID":"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f","Type":"ContainerStarted","Data":"8cea8501c6d3d21df7368a9b478e32e9f1d9875c2f5ebd237671d757531da604"}
Jul 18 16:41:20.687539 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:20.687474    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w95dc" event={"ID":"0728aabe-3077-4396-98c2-f6f7c1ec84db","Type":"ContainerStarted","Data":"2b0651bcc2f5c9f4be2159ef5755e3d42d6e177bc8a1f4348620a19dda8cc477"}
Jul 18 16:41:20.705813 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:20.705766    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w95dc" podStartSLOduration=33.745319905 podStartE2EDuration="45.705747656s" totalImagesPullingTime="11.960427751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 16:40:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:41:20.704403075 +0000 UTC m=+299.402233760" watchObservedRunningTime="2026-07-18 16:41:20.705747656 +0000 UTC m=+299.403578340"
Jul 18 16:41:21.259415 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:21.259385    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:21.688384 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:21.688353    2554 kubelet.go:1648] "Image garbage collection succeeded"
Jul 18 16:41:21.691682 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:21.691654    2554 generic.go:356] "Generic (PLEG): container finished" podID="57df3c7d-e80d-49b0-b1d0-4d21cbfc519f" containerID="2942c12209669484938a7a7124b3b1efb9d184eaacdb91a16513f6f692d30570" exitCode=0
Jul 18 16:41:21.691821 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:21.691737    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mwgzb" event={"ID":"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f","Type":"ContainerDied","Data":"2942c12209669484938a7a7124b3b1efb9d184eaacdb91a16513f6f692d30570"}
Jul 18 16:41:22.298769 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:22.298725    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:22.697460 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:22.697424    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mwgzb" event={"ID":"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f","Type":"ContainerStarted","Data":"c6870364ebd086f47c0dbddb5adaa84fe783b23c4c3ad45a9696d3624cdd592b"}
Jul 18 16:41:22.697664 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:22.697507    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mwgzb" event={"ID":"57df3c7d-e80d-49b0-b1d0-4d21cbfc519f","Type":"ContainerStarted","Data":"5366d40c514050e666a9b9cbf0a948a8dab4f5345593d23df86796a6875953e8"}
Jul 18 16:41:22.717096 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:22.717043    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mwgzb" podStartSLOduration=3.796042221 podStartE2EDuration="4.717027425s" totalImagesPullingTime="851.994616ms" totalInitContainerRuntime="68.990588ms" isStatefulPod=true podCreationTimestamp="2026-07-18 16:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:41:22.71544075 +0000 UTC m=+301.413271436" watchObservedRunningTime="2026-07-18 16:41:22.717027425 +0000 UTC m=+301.414858109"
Jul 18 16:41:22.749274 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:22.749239    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6947f4bd9f-px842" podUID="26cc0844-d905-444b-bea5-d6b4a2b148a2" containerName="registry" containerID="cri-o://89384b111304be2d66221da49d02c511244377592900a15f994acd5aabddbd5e" gracePeriod=30
Jul 18 16:41:22.899645 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:22.899613    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:22.987008 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:22.986978    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:41:22.999745 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:22.999712    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:23.075790 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.075752    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:23.091357 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.091333    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242"]
Jul 18 16:41:23.091585 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.091574    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26cc0844-d905-444b-bea5-d6b4a2b148a2" containerName="registry"
Jul 18 16:41:23.091631 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.091587    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="26cc0844-d905-444b-bea5-d6b4a2b148a2" containerName="registry"
Jul 18 16:41:23.091631 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.091629    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="26cc0844-d905-444b-bea5-d6b4a2b148a2" containerName="registry"
Jul 18 16:41:23.094233 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.094215    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242"
Jul 18 16:41:23.096482 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.096466    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5vwb6\""
Jul 18 16:41:23.096546 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.096501    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 18 16:41:23.103319 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.103295    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242"]
Jul 18 16:41:23.130298 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.130276    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:23.135290 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.135275    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-bound-sa-token\") pod \"26cc0844-d905-444b-bea5-d6b4a2b148a2\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") "
Jul 18 16:41:23.135376 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.135308    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls\") pod \"26cc0844-d905-444b-bea5-d6b4a2b148a2\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") "
Jul 18 16:41:23.135376 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.135355    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26cc0844-d905-444b-bea5-d6b4a2b148a2-installation-pull-secrets\") pod \"26cc0844-d905-444b-bea5-d6b4a2b148a2\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") "
Jul 18 16:41:23.135466 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.135390    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-certificates\") pod \"26cc0844-d905-444b-bea5-d6b4a2b148a2\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") "
Jul 18 16:41:23.135466 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.135408    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26cc0844-d905-444b-bea5-d6b4a2b148a2-trusted-ca\") pod \"26cc0844-d905-444b-bea5-d6b4a2b148a2\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") "
Jul 18 16:41:23.135466 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.135448    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26cc0844-d905-444b-bea5-d6b4a2b148a2-image-registry-private-configuration\") pod \"26cc0844-d905-444b-bea5-d6b4a2b148a2\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") "
Jul 18 16:41:23.135609 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.135487    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcnjv\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-kube-api-access-lcnjv\") pod \"26cc0844-d905-444b-bea5-d6b4a2b148a2\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") "
Jul 18 16:41:23.135609 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.135515    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26cc0844-d905-444b-bea5-d6b4a2b148a2-ca-trust-extracted\") pod \"26cc0844-d905-444b-bea5-d6b4a2b148a2\" (UID: \"26cc0844-d905-444b-bea5-d6b4a2b148a2\") "
Jul 18 16:41:23.136050 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.135863    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "26cc0844-d905-444b-bea5-d6b4a2b148a2" (UID: "26cc0844-d905-444b-bea5-d6b4a2b148a2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 16:41:23.136050 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.136001    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cc0844-d905-444b-bea5-d6b4a2b148a2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "26cc0844-d905-444b-bea5-d6b4a2b148a2" (UID: "26cc0844-d905-444b-bea5-d6b4a2b148a2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 16:41:23.138113 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.138092    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "26cc0844-d905-444b-bea5-d6b4a2b148a2" (UID: "26cc0844-d905-444b-bea5-d6b4a2b148a2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 16:41:23.138250 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.138187    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-kube-api-access-lcnjv" (OuterVolumeSpecName: "kube-api-access-lcnjv") pod "26cc0844-d905-444b-bea5-d6b4a2b148a2" (UID: "26cc0844-d905-444b-bea5-d6b4a2b148a2"). InnerVolumeSpecName "kube-api-access-lcnjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 16:41:23.138353 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.138257    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26cc0844-d905-444b-bea5-d6b4a2b148a2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "26cc0844-d905-444b-bea5-d6b4a2b148a2" (UID: "26cc0844-d905-444b-bea5-d6b4a2b148a2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 16:41:23.138353 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.138272    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26cc0844-d905-444b-bea5-d6b4a2b148a2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "26cc0844-d905-444b-bea5-d6b4a2b148a2" (UID: "26cc0844-d905-444b-bea5-d6b4a2b148a2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 16:41:23.138353 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.138312    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "26cc0844-d905-444b-bea5-d6b4a2b148a2" (UID: "26cc0844-d905-444b-bea5-d6b4a2b148a2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 16:41:23.144927 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.144907    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26cc0844-d905-444b-bea5-d6b4a2b148a2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "26cc0844-d905-444b-bea5-d6b4a2b148a2" (UID: "26cc0844-d905-444b-bea5-d6b4a2b148a2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 16:41:23.236879 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.236798    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f26b4eeb-fd35-4a94-beba-9b08799df449-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-p8242\" (UID: \"f26b4eeb-fd35-4a94-beba-9b08799df449\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242"
Jul 18 16:41:23.236879 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.236838    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbvdk\" (UniqueName: \"kubernetes.io/projected/f26b4eeb-fd35-4a94-beba-9b08799df449-kube-api-access-zbvdk\") pod \"monitoring-plugin-7b7b7bc8bf-p8242\" (UID: \"f26b4eeb-fd35-4a94-beba-9b08799df449\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242"
Jul 18 16:41:23.237023 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.236949    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26cc0844-d905-444b-bea5-d6b4a2b148a2-trusted-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:41:23.237023 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.236961    2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26cc0844-d905-444b-bea5-d6b4a2b148a2-image-registry-private-configuration\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:41:23.237023 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.236970    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcnjv\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-kube-api-access-lcnjv\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:41:23.237023 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.236979    2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26cc0844-d905-444b-bea5-d6b4a2b148a2-ca-trust-extracted\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:41:23.237023 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.236987    2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-bound-sa-token\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:41:23.237023 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.236995    2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:41:23.237023 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.237003    2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26cc0844-d905-444b-bea5-d6b4a2b148a2-installation-pull-secrets\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:41:23.237023 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.237010    2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26cc0844-d905-444b-bea5-d6b4a2b148a2-registry-certificates\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:41:23.337607 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.337571    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbvdk\" (UniqueName: \"kubernetes.io/projected/f26b4eeb-fd35-4a94-beba-9b08799df449-kube-api-access-zbvdk\") pod \"monitoring-plugin-7b7b7bc8bf-p8242\" (UID: \"f26b4eeb-fd35-4a94-beba-9b08799df449\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242"
Jul 18 16:41:23.338015 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.337651    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f26b4eeb-fd35-4a94-beba-9b08799df449-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-p8242\" (UID: \"f26b4eeb-fd35-4a94-beba-9b08799df449\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242"
Jul 18 16:41:23.338015 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:41:23.337737    2554 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 18 16:41:23.338015 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:41:23.337798    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26b4eeb-fd35-4a94-beba-9b08799df449-monitoring-plugin-cert podName:f26b4eeb-fd35-4a94-beba-9b08799df449 nodeName:}" failed. No retries permitted until 2026-07-18 16:41:23.837783684 +0000 UTC m=+302.535614346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/f26b4eeb-fd35-4a94-beba-9b08799df449-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-p8242" (UID: "f26b4eeb-fd35-4a94-beba-9b08799df449") : secret "monitoring-plugin-cert" not found
Jul 18 16:41:23.345906 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.345888    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbvdk\" (UniqueName: \"kubernetes.io/projected/f26b4eeb-fd35-4a94-beba-9b08799df449-kube-api-access-zbvdk\") pod \"monitoring-plugin-7b7b7bc8bf-p8242\" (UID: \"f26b4eeb-fd35-4a94-beba-9b08799df449\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242"
Jul 18 16:41:23.616491 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.616449    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:23.701940 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.701903    2554 generic.go:356] "Generic (PLEG): container finished" podID="26cc0844-d905-444b-bea5-d6b4a2b148a2" containerID="89384b111304be2d66221da49d02c511244377592900a15f994acd5aabddbd5e" exitCode=0
Jul 18 16:41:23.701940 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.701933    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6947f4bd9f-px842" event={"ID":"26cc0844-d905-444b-bea5-d6b4a2b148a2","Type":"ContainerDied","Data":"89384b111304be2d66221da49d02c511244377592900a15f994acd5aabddbd5e"}
Jul 18 16:41:23.702169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.701953    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6947f4bd9f-px842"
Jul 18 16:41:23.702169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.701987    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6947f4bd9f-px842" event={"ID":"26cc0844-d905-444b-bea5-d6b4a2b148a2","Type":"ContainerDied","Data":"79f8309be19a257b7c6ba924ffabad657ce7a65273c3d55c5d18cb11795a7ccc"}
Jul 18 16:41:23.702169 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.702007    2554 scope.go:117] "RemoveContainer" containerID="89384b111304be2d66221da49d02c511244377592900a15f994acd5aabddbd5e"
Jul 18 16:41:23.710534 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.710515    2554 scope.go:117] "RemoveContainer" containerID="89384b111304be2d66221da49d02c511244377592900a15f994acd5aabddbd5e"
Jul 18 16:41:23.710796 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:41:23.710778    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89384b111304be2d66221da49d02c511244377592900a15f994acd5aabddbd5e\": container with ID starting with 89384b111304be2d66221da49d02c511244377592900a15f994acd5aabddbd5e not found: ID does not exist" containerID="89384b111304be2d66221da49d02c511244377592900a15f994acd5aabddbd5e"
Jul 18 16:41:23.710880 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.710805    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89384b111304be2d66221da49d02c511244377592900a15f994acd5aabddbd5e"} err="failed to get container status \"89384b111304be2d66221da49d02c511244377592900a15f994acd5aabddbd5e\": rpc error: code = NotFound desc = could not find container \"89384b111304be2d66221da49d02c511244377592900a15f994acd5aabddbd5e\": container with ID starting with 89384b111304be2d66221da49d02c511244377592900a15f994acd5aabddbd5e not found: ID does not exist"
Jul 18 16:41:23.722107 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.722086    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6947f4bd9f-px842"]
Jul 18 16:41:23.725241 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.725222    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6947f4bd9f-px842"]
Jul 18 16:41:23.765504 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.765482    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26cc0844-d905-444b-bea5-d6b4a2b148a2" path="/var/lib/kubelet/pods/26cc0844-d905-444b-bea5-d6b4a2b148a2/volumes"
Jul 18 16:41:23.841170 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.841134    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f26b4eeb-fd35-4a94-beba-9b08799df449-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-p8242\" (UID: \"f26b4eeb-fd35-4a94-beba-9b08799df449\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242"
Jul 18 16:41:23.843565 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:23.843535    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f26b4eeb-fd35-4a94-beba-9b08799df449-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-p8242\" (UID: \"f26b4eeb-fd35-4a94-beba-9b08799df449\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242"
Jul 18 16:41:24.003079 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.002986    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242"
Jul 18 16:41:24.129907 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.129857    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242"]
Jul 18 16:41:24.132306 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:41:24.132281    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26b4eeb_fd35_4a94_beba_9b08799df449.slice/crio-14d8e37a480f13c7bc0c287e6ed44e58cdc44dddde98a1638f8f86aa3a1e559e WatchSource:0}: Error finding container 14d8e37a480f13c7bc0c287e6ed44e58cdc44dddde98a1638f8f86aa3a1e559e: Status 404 returned error can't find the container with id 14d8e37a480f13c7bc0c287e6ed44e58cdc44dddde98a1638f8f86aa3a1e559e
Jul 18 16:41:24.134317 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.134299    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 16:41:24.508968 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.508937    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 16:41:24.513166 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.513136    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:24.514206 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.514191    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.518387 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.518360    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 18 16:41:24.518569 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.518534    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bjutd9s13tftv\""
Jul 18 16:41:24.518971 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.518884    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 18 16:41:24.519208 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.519192    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 18 16:41:24.519342 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.519321    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6c5lw\""
Jul 18 16:41:24.519428 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.519371    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 18 16:41:24.519428 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.519221    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 18 16:41:24.519526 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.519321    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 18 16:41:24.519526 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.519195    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 18 16:41:24.519526 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.519377    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 18 16:41:24.519690 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.519256    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 18 16:41:24.519690 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.519647    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 18 16:41:24.519813 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.519791    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 18 16:41:24.519913 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.519895    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 18 16:41:24.520224 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.519904    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 18 16:41:24.520224 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.520009    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 18 16:41:24.521806 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.521786    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 18 16:41:24.530266 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.530243    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 16:41:24.650228 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650185    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650228 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650231    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-web-config\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650485 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650271    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650485 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650320    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650485 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650363    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650485 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650422    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650485 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650466    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-config\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650915 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650506    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650915 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650527    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650915 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650565    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b3a9861-4f13-4a63-baa1-b87b348da0b1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650915 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650600    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650915 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650627    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j9xq\" (UniqueName: \"kubernetes.io/projected/2b3a9861-4f13-4a63-baa1-b87b348da0b1-kube-api-access-8j9xq\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650915 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650654    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650915 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650683    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650915 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650707    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650915 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650731    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650915 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650782    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650915 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650808    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.650915 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650897    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.651530 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.650927    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b3a9861-4f13-4a63-baa1-b87b348da0b1-config-out\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.709031 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.708991    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242" event={"ID":"f26b4eeb-fd35-4a94-beba-9b08799df449","Type":"ContainerStarted","Data":"14d8e37a480f13c7bc0c287e6ed44e58cdc44dddde98a1638f8f86aa3a1e559e"}
Jul 18 16:41:24.752205 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.752172    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.752376 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.752216    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j9xq\" (UniqueName: \"kubernetes.io/projected/2b3a9861-4f13-4a63-baa1-b87b348da0b1-kube-api-access-8j9xq\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.752376 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.752248    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.752376 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.752276    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.752376 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.752298    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.752660 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.752631    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.752716 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.752668    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.752716 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.752697    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.752815 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.752731    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.752815 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.752787    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.752950 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.752819    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b3a9861-4f13-4a63-baa1-b87b348da0b1-config-out\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.752950 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.752892    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.752950 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.752920    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-web-config\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.753052 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.752951    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.753052 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.752980    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.753052 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.753010    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.753176 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.753027    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.753176 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.753136    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.753549 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.753458    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.754699 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.753884    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.754699 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.753940    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.754699 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.753979    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-config\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.754699 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.754010    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.754699 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.754035    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.754699 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.754067    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b3a9861-4f13-4a63-baa1-b87b348da0b1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.754699 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.754405    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.757443 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.756256    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.757443 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.756607    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.757443 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.757392    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.758061 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.758036    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.758609 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.758580    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.760713 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.760607    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b3a9861-4f13-4a63-baa1-b87b348da0b1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.760713 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.760646    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-config\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.761720 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.761407    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b3a9861-4f13-4a63-baa1-b87b348da0b1-config-out\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.761720 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.761520    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.761720 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.761537    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.762505 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.762458    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.762994 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.762972    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-web-config\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.764537 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.764495    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.765235 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.765216    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j9xq\" (UniqueName: \"kubernetes.io/projected/2b3a9861-4f13-4a63-baa1-b87b348da0b1-kube-api-access-8j9xq\") pod \"prometheus-k8s-0\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.826056 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.826023    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:24.985873 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:24.985618    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 16:41:24.989135 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:41:24.989094    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3a9861_4f13_4a63_baa1_b87b348da0b1.slice/crio-8d347ff62c3568cef7c43925e1ecbdba144a3028aaa0b7f84626c59a8dcb4712 WatchSource:0}: Error finding container 8d347ff62c3568cef7c43925e1ecbdba144a3028aaa0b7f84626c59a8dcb4712: Status 404 returned error can't find the container with id 8d347ff62c3568cef7c43925e1ecbdba144a3028aaa0b7f84626c59a8dcb4712
Jul 18 16:41:25.714718 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:25.714253    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242" event={"ID":"f26b4eeb-fd35-4a94-beba-9b08799df449","Type":"ContainerStarted","Data":"d7ce4fbff1a0969573666e314da5435ece338322816a8316bea913e199783c88"}
Jul 18 16:41:25.714718 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:25.714318    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242"
Jul 18 16:41:25.715638 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:25.715609    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b3a9861-4f13-4a63-baa1-b87b348da0b1","Type":"ContainerStarted","Data":"8d347ff62c3568cef7c43925e1ecbdba144a3028aaa0b7f84626c59a8dcb4712"}
Jul 18 16:41:25.719894 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:25.719870    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242"
Jul 18 16:41:25.743181 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:25.743130    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p8242" podStartSLOduration=1.414228616 podStartE2EDuration="2.743111142s" totalImagesPullingTime="1.328882526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:41:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:41:25.728341559 +0000 UTC m=+304.426172243" watchObservedRunningTime="2026-07-18 16:41:25.743111142 +0000 UTC m=+304.440941827"
Jul 18 16:41:26.720770 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:26.720737    2554 generic.go:356] "Generic (PLEG): container finished" podID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerID="b44d8bc9ec942d0bcf0c22745c697d50f1f2f3b4e64939ab7f9493c6cd59b11c" exitCode=0
Jul 18 16:41:26.721162 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:26.720770    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b3a9861-4f13-4a63-baa1-b87b348da0b1","Type":"ContainerDied","Data":"b44d8bc9ec942d0bcf0c22745c697d50f1f2f3b4e64939ab7f9493c6cd59b11c"}
Jul 18 16:41:28.094902 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:28.094867    2554 patch_prober.go:28] interesting pod/image-registry-7dcbf79b9c-x7qxx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 18 16:41:28.095313 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:28.094927    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx" podUID="95484bc2-9453-4837-8f68-45a375d918bf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 18 16:41:28.609329 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:28.609285    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:28.692578 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:28.692525    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:28.756810 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:28.756780    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:28.857237 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:28.857204    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:28.958536 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:28.958410    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:29.245680 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:29.245600    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:29.346769 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:29.346736    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:29.592908 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:29.592882    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7dcbf79b9c-x7qxx"
Jul 18 16:41:29.670826 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:29.670789    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:29.725354 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:29.725307    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:30.042863 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:30.042822    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:30.144712 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:30.144680    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:30.465359 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:30.465327    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:30.657829 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:30.657798    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:30.738835 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:30.738755    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b3a9861-4f13-4a63-baa1-b87b348da0b1","Type":"ContainerStarted","Data":"5a546995ed002496448f204ecfb03b16c39f125f8ae162c89b0db6c8400ef0fe"}
Jul 18 16:41:30.738835 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:30.738826    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b3a9861-4f13-4a63-baa1-b87b348da0b1","Type":"ContainerStarted","Data":"2b55260ba62ae817a1a213b09566b76ae1fbbdfa8108412d7567fec2cd06763c"}
Jul 18 16:41:30.759958 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:30.759929    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:41:31.028093 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:31.028013    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:31.476226 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:31.476187    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:31.574164 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:31.574128    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:32.274886 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:32.274817    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:32.751248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:32.751215    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b3a9861-4f13-4a63-baa1-b87b348da0b1","Type":"ContainerStarted","Data":"d878970ef1e6b177d4e7b38aedb72b8716bd1dcd016d90ac1c1c382c8334657f"}
Jul 18 16:41:32.751589 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:32.751301    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b3a9861-4f13-4a63-baa1-b87b348da0b1","Type":"ContainerStarted","Data":"b8f9106ffc1750f5c14ae90729b2025eed950282ffb63a9138ba95c793656a1a"}
Jul 18 16:41:32.751589 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:32.751327    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b3a9861-4f13-4a63-baa1-b87b348da0b1","Type":"ContainerStarted","Data":"b6c433da7b63023607c4b62d353eba17ca1fa8073d38f74255459277899cb280"}
Jul 18 16:41:32.751589 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:32.751358    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b3a9861-4f13-4a63-baa1-b87b348da0b1","Type":"ContainerStarted","Data":"12a54cc1437ef792536cd86f08533a1d0e0b4a55daf88aef75471cf9396cbfc5"}
Jul 18 16:41:32.780254 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:32.780216    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.8401986900000002 podStartE2EDuration="8.780202941s" totalImagesPullingTime="5.921288874s" totalInitContainerRuntime="18.715377ms" isStatefulPod=false podCreationTimestamp="2026-07-18 16:41:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:41:32.778734794 +0000 UTC m=+311.476565477" watchObservedRunningTime="2026-07-18 16:41:32.780202941 +0000 UTC m=+311.478033667"
Jul 18 16:41:33.484562 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:33.484529    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:33.585817 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:33.585773    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:34.826479 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:34.826435    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:41:36.106059 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:36.106027    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:36.162593 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:36.162549    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:38.476942 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:38.476913    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:38.578113 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:38.578059    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:41.303083 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:41.303055    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:41.402613 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:41.402570    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:42.560530 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:42.560498    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:42.659016 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:42.658980    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:45.274179 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:45.274144    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:45.334363 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:45.334331    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:45.506937 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:45.506906    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:45.606192 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:45.606159    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:46.771163 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:46.771133    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:46.868203 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:46.868172    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:49.876180 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:49.876148    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:55.833110 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:55.833079    2554 generic.go:356] "Generic (PLEG): container finished" podID="05af46b3-3c72-416b-b837-d9412910c1b5" containerID="a770cdfde263c11cfc6154dca7dcb9ea2cfbdf8b70efedf58167652416887444" exitCode=0
Jul 18 16:41:55.833477 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:55.833122    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kvsvt" event={"ID":"05af46b3-3c72-416b-b837-d9412910c1b5","Type":"ContainerDied","Data":"a770cdfde263c11cfc6154dca7dcb9ea2cfbdf8b70efedf58167652416887444"}
Jul 18 16:41:55.833477 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:55.833450    2554 scope.go:117] "RemoveContainer" containerID="a770cdfde263c11cfc6154dca7dcb9ea2cfbdf8b70efedf58167652416887444"
Jul 18 16:41:56.839022 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:56.838989    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kvsvt" event={"ID":"05af46b3-3c72-416b-b837-d9412910c1b5","Type":"ContainerStarted","Data":"cf3845aa419913c372dd77a08d1adedc2f6eb187db3bfd67c8d04128f8765198"}
Jul 18 16:41:57.088793 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:57.088761    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:41:57.187555 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:41:57.187459    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:00.503958 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:00.503927    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:00.556897 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:00.556862    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:01.685124 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:01.685070    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:01.733798 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:01.733763    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:02.479870 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:02.479820    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:02.530278 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:02.530248    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:04.359451 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:04.359420    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:04.417548 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:04.417510    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:15.503519 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:15.503436    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:15.561089 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:15.561063    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:24.827028 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:24.826987    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:24.846211 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:24.846182    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:24.953392 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:24.953366    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:35.627725 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:35.627681    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:42:35.630122 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:35.630101    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9ff934-e093-4adb-b818-279e624555de-metrics-certs\") pod \"network-metrics-daemon-hczmz\" (UID: \"2f9ff934-e093-4adb-b818-279e624555de\") " pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:42:35.863038 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:35.863003    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pm4pn\""
Jul 18 16:42:35.870983 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:35.870958    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hczmz"
Jul 18 16:42:36.008389 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:36.008228    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hczmz"]
Jul 18 16:42:36.010879 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:42:36.010835    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9ff934_e093_4adb_b818_279e624555de.slice/crio-38be6789cc67c0fc044e58046a84390845beb045b6e690874e0d3db8fa5cedbe WatchSource:0}: Error finding container 38be6789cc67c0fc044e58046a84390845beb045b6e690874e0d3db8fa5cedbe: Status 404 returned error can't find the container with id 38be6789cc67c0fc044e58046a84390845beb045b6e690874e0d3db8fa5cedbe
Jul 18 16:42:36.077370 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:36.077334    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:36.130248 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:36.130223    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:36.981230 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:36.981203    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hczmz" event={"ID":"2f9ff934-e093-4adb-b818-279e624555de","Type":"ContainerStarted","Data":"39d3797e577643c7ab002b2f5afca67db7f5b2ed017bc035f1f740a5bdcf8c56"}
Jul 18 16:42:36.981563 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:36.981250    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hczmz" event={"ID":"2f9ff934-e093-4adb-b818-279e624555de","Type":"ContainerStarted","Data":"a0cb3f1f8328d7ecf97e283e98360284e9c09d0decde5521d40470979cddd12a"}
Jul 18 16:42:36.981563 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:36.981267    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hczmz" event={"ID":"2f9ff934-e093-4adb-b818-279e624555de","Type":"ContainerStarted","Data":"38be6789cc67c0fc044e58046a84390845beb045b6e690874e0d3db8fa5cedbe"}
Jul 18 16:42:36.999069 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:36.999026    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hczmz" podStartSLOduration=374.225837714 podStartE2EDuration="6m14.999012872s" totalImagesPullingTime="773.175158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:42:36.997883098 +0000 UTC m=+375.695713783" watchObservedRunningTime="2026-07-18 16:42:36.999012872 +0000 UTC m=+375.696843556"
Jul 18 16:42:37.088463 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:37.088433    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:37.189146 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:37.189117    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:37.508380 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:37.508267    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:37.710435 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:37.710380    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:42.875710 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:42.875671    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 16:42:42.876309 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:42.876035    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="prometheus" containerID="cri-o://2b55260ba62ae817a1a213b09566b76ae1fbbdfa8108412d7567fec2cd06763c" gracePeriod=600
Jul 18 16:42:42.876428 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:42.876398    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d878970ef1e6b177d4e7b38aedb72b8716bd1dcd016d90ac1c1c382c8334657f" gracePeriod=600
Jul 18 16:42:42.876534 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:42.876512    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="kube-rbac-proxy" containerID="cri-o://b8f9106ffc1750f5c14ae90729b2025eed950282ffb63a9138ba95c793656a1a" gracePeriod=600
Jul 18 16:42:42.877996 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:42.876736    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="thanos-sidecar" containerID="cri-o://12a54cc1437ef792536cd86f08533a1d0e0b4a55daf88aef75471cf9396cbfc5" gracePeriod=600
Jul 18 16:42:42.877996 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:42.876764    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="kube-rbac-proxy-web" containerID="cri-o://b6c433da7b63023607c4b62d353eba17ca1fa8073d38f74255459277899cb280" gracePeriod=600
Jul 18 16:42:42.878145 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:42.876781    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="config-reloader" containerID="cri-o://5a546995ed002496448f204ecfb03b16c39f125f8ae162c89b0db6c8400ef0fe" gracePeriod=600
Jul 18 16:42:43.007978 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.007951    2554 generic.go:356] "Generic (PLEG): container finished" podID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerID="d878970ef1e6b177d4e7b38aedb72b8716bd1dcd016d90ac1c1c382c8334657f" exitCode=0
Jul 18 16:42:43.007978 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.007977    2554 generic.go:356] "Generic (PLEG): container finished" podID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerID="b8f9106ffc1750f5c14ae90729b2025eed950282ffb63a9138ba95c793656a1a" exitCode=0
Jul 18 16:42:43.007978 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.007985    2554 generic.go:356] "Generic (PLEG): container finished" podID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerID="b6c433da7b63023607c4b62d353eba17ca1fa8073d38f74255459277899cb280" exitCode=0
Jul 18 16:42:43.007978 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.007992    2554 generic.go:356] "Generic (PLEG): container finished" podID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerID="12a54cc1437ef792536cd86f08533a1d0e0b4a55daf88aef75471cf9396cbfc5" exitCode=0
Jul 18 16:42:43.008215 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.008000    2554 generic.go:356] "Generic (PLEG): container finished" podID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerID="5a546995ed002496448f204ecfb03b16c39f125f8ae162c89b0db6c8400ef0fe" exitCode=0
Jul 18 16:42:43.008215 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.008008    2554 generic.go:356] "Generic (PLEG): container finished" podID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerID="2b55260ba62ae817a1a213b09566b76ae1fbbdfa8108412d7567fec2cd06763c" exitCode=0
Jul 18 16:42:43.008215 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.008030    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b3a9861-4f13-4a63-baa1-b87b348da0b1","Type":"ContainerDied","Data":"d878970ef1e6b177d4e7b38aedb72b8716bd1dcd016d90ac1c1c382c8334657f"}
Jul 18 16:42:43.008215 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.008121    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b3a9861-4f13-4a63-baa1-b87b348da0b1","Type":"ContainerDied","Data":"b8f9106ffc1750f5c14ae90729b2025eed950282ffb63a9138ba95c793656a1a"}
Jul 18 16:42:43.008215 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.008164    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b3a9861-4f13-4a63-baa1-b87b348da0b1","Type":"ContainerDied","Data":"b6c433da7b63023607c4b62d353eba17ca1fa8073d38f74255459277899cb280"}
Jul 18 16:42:43.008215 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.008206    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b3a9861-4f13-4a63-baa1-b87b348da0b1","Type":"ContainerDied","Data":"12a54cc1437ef792536cd86f08533a1d0e0b4a55daf88aef75471cf9396cbfc5"}
Jul 18 16:42:43.008387 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.008281    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b3a9861-4f13-4a63-baa1-b87b348da0b1","Type":"ContainerDied","Data":"5a546995ed002496448f204ecfb03b16c39f125f8ae162c89b0db6c8400ef0fe"}
Jul 18 16:42:43.008387 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.008320    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b3a9861-4f13-4a63-baa1-b87b348da0b1","Type":"ContainerDied","Data":"2b55260ba62ae817a1a213b09566b76ae1fbbdfa8108412d7567fec2cd06763c"}
Jul 18 16:42:43.130880 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.130803    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:43.188491 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.188447    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b3a9861-4f13-4a63-baa1-b87b348da0b1-tls-assets\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.188673 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.188501    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-tls\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.188673 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.188531    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-metrics-client-ca\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.188673 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.188558    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-thanos-prometheus-http-client-file\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.188673 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.188590    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-serving-certs-ca-bundle\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.188673 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.188622    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-metrics-client-certs\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.188673 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.188656    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-2\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.189037 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.188684    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-web-config\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.189037 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.188723    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-config\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.189037 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.188776    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-0\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.189037 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.188807    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.189037 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.188833    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-db\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.190768 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.189428    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.190768 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.189517    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-trusted-ca-bundle\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.190768 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.189550    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-grpc-tls\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.190768 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.189577    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b3a9861-4f13-4a63-baa1-b87b348da0b1-config-out\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.190768 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.189607    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j9xq\" (UniqueName: \"kubernetes.io/projected/2b3a9861-4f13-4a63-baa1-b87b348da0b1-kube-api-access-8j9xq\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.190768 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.189646    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-1\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.190768 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.189689    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-kube-rbac-proxy\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.190768 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.189715    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-kubelet-serving-ca-bundle\") pod \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\" (UID: \"2b3a9861-4f13-4a63-baa1-b87b348da0b1\") "
Jul 18 16:42:43.190768 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.190371    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 16:42:43.190768 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.190459    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 16:42:43.190768 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.190498    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 16:42:43.191497 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.191465    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 16:42:43.191918 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.191890    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 16:42:43.194888 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.194711    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 16:42:43.194888 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.194793    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b3a9861-4f13-4a63-baa1-b87b348da0b1-config-out" (OuterVolumeSpecName: "config-out") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 16:42:43.194888 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.194874    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3a9861-4f13-4a63-baa1-b87b348da0b1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 16:42:43.195203 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.195161    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 16:42:43.195203 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.195180    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 16:42:43.195532 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.195507    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 16:42:43.196079 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.196055    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3a9861-4f13-4a63-baa1-b87b348da0b1-kube-api-access-8j9xq" (OuterVolumeSpecName: "kube-api-access-8j9xq") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "kube-api-access-8j9xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 16:42:43.196433 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.196393    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 16:42:43.196433 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.196412    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 16:42:43.197258 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.197215    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-config" (OuterVolumeSpecName: "config") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 16:42:43.197446 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.197425    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 16:42:43.197562 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.197496    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 16:42:43.197911 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.197880    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 16:42:43.198268 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.198242    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 16:42:43.205813 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.205793    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-web-config" (OuterVolumeSpecName: "web-config") pod "2b3a9861-4f13-4a63-baa1-b87b348da0b1" (UID: "2b3a9861-4f13-4a63-baa1-b87b348da0b1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 16:42:43.290506 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290472    2554 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-kube-rbac-proxy\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290506 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290503    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290506 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290514    2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b3a9861-4f13-4a63-baa1-b87b348da0b1-tls-assets\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290523    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290532    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-metrics-client-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290541    2554 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290549    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290558    2554 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-metrics-client-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290567    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290576    2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-web-config\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290583    2554 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-config\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290592    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290600    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290609    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-db\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290617    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290625    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290633    2554 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2b3a9861-4f13-4a63-baa1-b87b348da0b1-secret-grpc-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290640    2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b3a9861-4f13-4a63-baa1-b87b348da0b1-config-out\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290648    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j9xq\" (UniqueName: \"kubernetes.io/projected/2b3a9861-4f13-4a63-baa1-b87b348da0b1-kube-api-access-8j9xq\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:43.290730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:43.290655    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2b3a9861-4f13-4a63-baa1-b87b348da0b1-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:42:44.015110 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.015073    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b3a9861-4f13-4a63-baa1-b87b348da0b1","Type":"ContainerDied","Data":"8d347ff62c3568cef7c43925e1ecbdba144a3028aaa0b7f84626c59a8dcb4712"}
Jul 18 16:42:44.015110 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.015095    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.015545 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.015150    2554 scope.go:117] "RemoveContainer" containerID="d878970ef1e6b177d4e7b38aedb72b8716bd1dcd016d90ac1c1c382c8334657f"
Jul 18 16:42:44.023510 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.023491    2554 scope.go:117] "RemoveContainer" containerID="b8f9106ffc1750f5c14ae90729b2025eed950282ffb63a9138ba95c793656a1a"
Jul 18 16:42:44.030214 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.030194    2554 scope.go:117] "RemoveContainer" containerID="b6c433da7b63023607c4b62d353eba17ca1fa8073d38f74255459277899cb280"
Jul 18 16:42:44.036714 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.036694    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 16:42:44.036958 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.036918    2554 scope.go:117] "RemoveContainer" containerID="12a54cc1437ef792536cd86f08533a1d0e0b4a55daf88aef75471cf9396cbfc5"
Jul 18 16:42:44.041368 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.041349    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 16:42:44.044138 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.044120    2554 scope.go:117] "RemoveContainer" containerID="5a546995ed002496448f204ecfb03b16c39f125f8ae162c89b0db6c8400ef0fe"
Jul 18 16:42:44.050554 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.050536    2554 scope.go:117] "RemoveContainer" containerID="2b55260ba62ae817a1a213b09566b76ae1fbbdfa8108412d7567fec2cd06763c"
Jul 18 16:42:44.057420 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.057396    2554 scope.go:117] "RemoveContainer" containerID="b44d8bc9ec942d0bcf0c22745c697d50f1f2f3b4e64939ab7f9493c6cd59b11c"
Jul 18 16:42:44.065555 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065535    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 16:42:44.065785 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065773    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="kube-rbac-proxy"
Jul 18 16:42:44.065831 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065788    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="kube-rbac-proxy"
Jul 18 16:42:44.065831 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065796    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="kube-rbac-proxy-thanos"
Jul 18 16:42:44.065831 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065801    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="kube-rbac-proxy-thanos"
Jul 18 16:42:44.065831 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065807    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="init-config-reloader"
Jul 18 16:42:44.065831 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065813    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="init-config-reloader"
Jul 18 16:42:44.065831 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065822    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="kube-rbac-proxy-web"
Jul 18 16:42:44.065831 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065828    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="kube-rbac-proxy-web"
Jul 18 16:42:44.066043 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065838    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="config-reloader"
Jul 18 16:42:44.066043 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065860    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="config-reloader"
Jul 18 16:42:44.066043 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065869    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="prometheus"
Jul 18 16:42:44.066043 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065874    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="prometheus"
Jul 18 16:42:44.066043 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065887    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="thanos-sidecar"
Jul 18 16:42:44.066043 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065893    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="thanos-sidecar"
Jul 18 16:42:44.066043 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065934    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="kube-rbac-proxy-thanos"
Jul 18 16:42:44.066043 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065941    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="prometheus"
Jul 18 16:42:44.066043 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065949    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="thanos-sidecar"
Jul 18 16:42:44.066043 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065955    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="config-reloader"
Jul 18 16:42:44.066043 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065960    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="kube-rbac-proxy-web"
Jul 18 16:42:44.066043 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.065965    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" containerName="kube-rbac-proxy"
Jul 18 16:42:44.071098 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.071077    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.074033 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.074010    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 18 16:42:44.074515 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.074499    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 18 16:42:44.074594 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.074501    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 18 16:42:44.075332 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.075313    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 18 16:42:44.075406 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.075322    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 18 16:42:44.075523 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.075506    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6c5lw\""
Jul 18 16:42:44.075588 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.075529    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 18 16:42:44.075637 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.075599    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 18 16:42:44.075691 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.075665    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 18 16:42:44.076372 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.076357    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bjutd9s13tftv\""
Jul 18 16:42:44.076728 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.076711    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 18 16:42:44.076962 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.076948    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 18 16:42:44.077070 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.077056    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 18 16:42:44.077144 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.077094    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 18 16:42:44.080492 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.080473    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 18 16:42:44.087519 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.087503    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 18 16:42:44.096307 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096091    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51e008b8-832b-4103-8d3a-8ecffd21ecc7-config-out\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096307 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096129    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096307 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096153    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096307 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096176    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096307 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096205    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096307 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096231    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096307 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096281    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096338    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096369    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096397    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096419    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-web-config\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096445    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-config\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096464    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/51e008b8-832b-4103-8d3a-8ecffd21ecc7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096494    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096528    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096558    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51e008b8-832b-4103-8d3a-8ecffd21ecc7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096581    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxm6f\" (UniqueName: \"kubernetes.io/projected/51e008b8-832b-4103-8d3a-8ecffd21ecc7-kube-api-access-qxm6f\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096611    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096636    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.096753 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.096710    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.098346 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.098324    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 18 16:42:44.098716 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.098698    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 16:42:44.198102 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.198057    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.198288 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.198117    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.198288 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.198165    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51e008b8-832b-4103-8d3a-8ecffd21ecc7-config-out\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.198288 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.198196    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.198288 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.198231    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.198288 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.198255    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.198288 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.198284    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.198557 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.198307    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.198557 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.198338    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.199410 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.199017    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.199410 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.199139    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.199410 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.199178    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.199410 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.199213    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.199410 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.199240    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-web-config\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.199410 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.199268    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-config\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.199410 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.199292    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/51e008b8-832b-4103-8d3a-8ecffd21ecc7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.199410 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.199303    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.199410 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.199312    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.199410 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.199326    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.199410 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.199378    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.199410 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.199384    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.200102 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.199427    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51e008b8-832b-4103-8d3a-8ecffd21ecc7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.200102 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.199454    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxm6f\" (UniqueName: \"kubernetes.io/projected/51e008b8-832b-4103-8d3a-8ecffd21ecc7-kube-api-access-qxm6f\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.200102 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.199483    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.201868 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.201574    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.201868 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.201764    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51e008b8-832b-4103-8d3a-8ecffd21ecc7-config-out\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.202291 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.202223    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.202291 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.202287    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.202618 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.202577    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51e008b8-832b-4103-8d3a-8ecffd21ecc7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.202721 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.202672    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/51e008b8-832b-4103-8d3a-8ecffd21ecc7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.203602 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.203560    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.203686 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.203603    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/51e008b8-832b-4103-8d3a-8ecffd21ecc7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.203754 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.203741    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.204002 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.203976    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.204349 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.204329    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-web-config\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.204598 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.204574    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.204682 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.204666    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-config\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.204924 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.204907    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.206170 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.206150    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/51e008b8-832b-4103-8d3a-8ecffd21ecc7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.207486 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.207471    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxm6f\" (UniqueName: \"kubernetes.io/projected/51e008b8-832b-4103-8d3a-8ecffd21ecc7-kube-api-access-qxm6f\") pod \"prometheus-k8s-0\" (UID: \"51e008b8-832b-4103-8d3a-8ecffd21ecc7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.381773 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.381743    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:44.518694 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:44.518670    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 18 16:42:44.520700 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:42:44.520676    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e008b8_832b_4103_8d3a_8ecffd21ecc7.slice/crio-2b4f2cf6e43746c2de90b0fcee9edd97574cc66141ceb958a2144b8cbff835b1 WatchSource:0}: Error finding container 2b4f2cf6e43746c2de90b0fcee9edd97574cc66141ceb958a2144b8cbff835b1: Status 404 returned error can't find the container with id 2b4f2cf6e43746c2de90b0fcee9edd97574cc66141ceb958a2144b8cbff835b1
Jul 18 16:42:45.020912 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:45.020882    2554 generic.go:356] "Generic (PLEG): container finished" podID="51e008b8-832b-4103-8d3a-8ecffd21ecc7" containerID="1dde42145ec5a644f98e605ec263f1a5cd59e3b169ab9d669677ec211772c72e" exitCode=0
Jul 18 16:42:45.021351 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:45.020974    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51e008b8-832b-4103-8d3a-8ecffd21ecc7","Type":"ContainerDied","Data":"1dde42145ec5a644f98e605ec263f1a5cd59e3b169ab9d669677ec211772c72e"}
Jul 18 16:42:45.021351 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:45.021043    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51e008b8-832b-4103-8d3a-8ecffd21ecc7","Type":"ContainerStarted","Data":"2b4f2cf6e43746c2de90b0fcee9edd97574cc66141ceb958a2144b8cbff835b1"}
Jul 18 16:42:45.047833 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:45.047804    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:45.765874 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:45.765816    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b3a9861-4f13-4a63-baa1-b87b348da0b1" path="/var/lib/kubelet/pods/2b3a9861-4f13-4a63-baa1-b87b348da0b1/volumes"
Jul 18 16:42:46.029630 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:46.029542    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51e008b8-832b-4103-8d3a-8ecffd21ecc7","Type":"ContainerStarted","Data":"68b3d1853eb759cef75589f14ac17547167e5d9eeea85a3dfe14979b4466f8a9"}
Jul 18 16:42:46.029630 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:46.029612    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51e008b8-832b-4103-8d3a-8ecffd21ecc7","Type":"ContainerStarted","Data":"90f608bbb5c142fdbe2e893ace707e1b79f8117e90d77c24aaa12dcd7de1d7d5"}
Jul 18 16:42:46.030059 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:46.029640    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51e008b8-832b-4103-8d3a-8ecffd21ecc7","Type":"ContainerStarted","Data":"12ddbb2cd1be077b19e836b20eb28e01f4b2a10418f1fad1256dc1b9e7c50586"}
Jul 18 16:42:46.030059 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:46.029668    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51e008b8-832b-4103-8d3a-8ecffd21ecc7","Type":"ContainerStarted","Data":"c3e849ac792ee649d949c48628dc4423356d321d871960756d8ebe15593cfefc"}
Jul 18 16:42:46.030059 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:46.029701    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51e008b8-832b-4103-8d3a-8ecffd21ecc7","Type":"ContainerStarted","Data":"e98a738c3b184a38dcb6b4e6baf90a412bbaaea70c1a0c3b56d3889d6c017528"}
Jul 18 16:42:46.030059 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:46.029725    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51e008b8-832b-4103-8d3a-8ecffd21ecc7","Type":"ContainerStarted","Data":"74e40fa2753f73cfb65cd833d75a8837f356ef145b162d6fa5b88466a8f1ce8d"}
Jul 18 16:42:46.057070 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:46.057017    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.032197963 podStartE2EDuration="2.057000851s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.802888ms" isStatefulPod=false podCreationTimestamp="2026-07-18 16:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:42:46.05524743 +0000 UTC m=+384.753078115" watchObservedRunningTime="2026-07-18 16:42:46.057000851 +0000 UTC m=+384.754831534"
Jul 18 16:42:46.151901 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:46.151867    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:46.203147 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:46.203113    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:49.382325 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:49.382289    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:42:51.623189 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:51.623152    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:51.722870 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:51.722833    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:57.198862 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:57.198814    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:57.251200 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:57.251171    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:42:57.301269 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:42:57.301242    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:43:30.530709 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:43:30.530680    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:43:30.582974 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:43:30.582939    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:43:44.382229 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:43:44.382132    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:43:44.397884 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:43:44.397839    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:43:45.253358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:43:45.253330    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 18 16:43:52.733019 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:43:52.732983    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:43:52.786632 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:43:52.786603    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:44:04.648971 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:44:04.648931    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:44:04.699342 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:44:04.699308    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:44:15.527719 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:44:15.527680    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:44:15.621809 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:44:15.621781    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:44:28.847808 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:44:28.847772    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:44:28.908113 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:44:28.908082    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:44:39.605807 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:44:39.605775    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:44:39.656372 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:44:39.656338    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:44:39.709364 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:44:39.709330    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:44:42.154929 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:44:42.154898    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:07.730545 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:07.730507    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:07.828567 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:07.828529    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:29.498100 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:29.498029    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:30.848109 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:30.848081    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:33.119760 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:33.119727    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:33.170826 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:33.170787    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:34.965629 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:34.965593    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:35.085068 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:35.085034    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:35.205855 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:35.205826    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:37.175334 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:37.175299    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:37.578094 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:37.578062    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:40.104425 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:40.104394    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:40.204602 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:40.204568    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:45.584800 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:45.584758    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:46.509202 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:46.509172    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:46.610035 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:46.610001    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:52.937211 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:52.937168    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:53.036986 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:53.036953    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:54.923771 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:54.923740    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:56.804772 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:56.804740    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:45:56.903699 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:45:56.903663    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:07.713132 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:07.713094    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:07.812702 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:07.812672    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:08.324545 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.324514    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:08.523592 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.523558    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:08.616080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.616049    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:08.663223 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.663193    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"]
Jul 18 16:46:08.666555 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.666539    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.669176 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.669149    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 18 16:46:08.669292 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.669151    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 18 16:46:08.669344 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.669324    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kffsd\""
Jul 18 16:46:08.669393 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.669372    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 18 16:46:08.669839 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.669824    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 18 16:46:08.680412 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.680390    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"]
Jul 18 16:46:08.714132 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.714104    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxjz7\" (UniqueName: \"kubernetes.io/projected/7af4d913-b0bd-4af3-92e2-2ae1ad9debf4-kube-api-access-zxjz7\") pod \"opendatahub-operator-controller-manager-c768ddb5d-whpnh\" (UID: \"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4\") " pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.714523 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.714165    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7af4d913-b0bd-4af3-92e2-2ae1ad9debf4-tmp\") pod \"opendatahub-operator-controller-manager-c768ddb5d-whpnh\" (UID: \"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4\") " pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.714523 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.714206    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7af4d913-b0bd-4af3-92e2-2ae1ad9debf4-webhook-cert\") pod \"opendatahub-operator-controller-manager-c768ddb5d-whpnh\" (UID: \"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4\") " pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.714636 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.714520    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7af4d913-b0bd-4af3-92e2-2ae1ad9debf4-manifests\") pod \"opendatahub-operator-controller-manager-c768ddb5d-whpnh\" (UID: \"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4\") " pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.714636 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.714559    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7af4d913-b0bd-4af3-92e2-2ae1ad9debf4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c768ddb5d-whpnh\" (UID: \"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4\") " pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.815406 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.815371    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjz7\" (UniqueName: \"kubernetes.io/projected/7af4d913-b0bd-4af3-92e2-2ae1ad9debf4-kube-api-access-zxjz7\") pod \"opendatahub-operator-controller-manager-c768ddb5d-whpnh\" (UID: \"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4\") " pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.815590 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.815426    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7af4d913-b0bd-4af3-92e2-2ae1ad9debf4-tmp\") pod \"opendatahub-operator-controller-manager-c768ddb5d-whpnh\" (UID: \"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4\") " pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.815590 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.815457    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7af4d913-b0bd-4af3-92e2-2ae1ad9debf4-webhook-cert\") pod \"opendatahub-operator-controller-manager-c768ddb5d-whpnh\" (UID: \"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4\") " pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.815590 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.815486    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7af4d913-b0bd-4af3-92e2-2ae1ad9debf4-manifests\") pod \"opendatahub-operator-controller-manager-c768ddb5d-whpnh\" (UID: \"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4\") " pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.815590 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.815509    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7af4d913-b0bd-4af3-92e2-2ae1ad9debf4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c768ddb5d-whpnh\" (UID: \"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4\") " pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.816005 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.815942    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7af4d913-b0bd-4af3-92e2-2ae1ad9debf4-manifests\") pod \"opendatahub-operator-controller-manager-c768ddb5d-whpnh\" (UID: \"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4\") " pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.817199 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.817174    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:08.818083 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.818065    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7af4d913-b0bd-4af3-92e2-2ae1ad9debf4-tmp\") pod \"opendatahub-operator-controller-manager-c768ddb5d-whpnh\" (UID: \"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4\") " pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.818166 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.818127    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7af4d913-b0bd-4af3-92e2-2ae1ad9debf4-webhook-cert\") pod \"opendatahub-operator-controller-manager-c768ddb5d-whpnh\" (UID: \"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4\") " pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.818388 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.818367    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7af4d913-b0bd-4af3-92e2-2ae1ad9debf4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c768ddb5d-whpnh\" (UID: \"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4\") " pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.823976 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.823952    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxjz7\" (UniqueName: \"kubernetes.io/projected/7af4d913-b0bd-4af3-92e2-2ae1ad9debf4-kube-api-access-zxjz7\") pod \"opendatahub-operator-controller-manager-c768ddb5d-whpnh\" (UID: \"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4\") " pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:08.914368 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.914293    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:08.976422 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:08.976396    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:09.011988 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:09.011935    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:09.088927 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:09.088896    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:09.112010 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:09.111991    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"]
Jul 18 16:46:09.115123 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:46:09.115098    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af4d913_b0bd_4af3_92e2_2ae1ad9debf4.slice/crio-d7b098a258a4ab1444eb744894aa79e6f21ef2909db71754deaebccce98b137c WatchSource:0}: Error finding container d7b098a258a4ab1444eb744894aa79e6f21ef2909db71754deaebccce98b137c: Status 404 returned error can't find the container with id d7b098a258a4ab1444eb744894aa79e6f21ef2909db71754deaebccce98b137c
Jul 18 16:46:09.156618 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:09.156592    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:09.269325 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:09.269253    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:09.368788 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:09.368760    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:09.469283 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:09.469251    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:09.772443 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:09.772409    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh" event={"ID":"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4","Type":"ContainerStarted","Data":"d7b098a258a4ab1444eb744894aa79e6f21ef2909db71754deaebccce98b137c"}
Jul 18 16:46:12.787476 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:12.787440    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh" event={"ID":"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4","Type":"ContainerStarted","Data":"e07012797319e730288b403cb408a47bdeea25667cf0d91db1fb306d6a3891f5"}
Jul 18 16:46:13.793462 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:13.793373    2554 generic.go:356] "Generic (PLEG): container finished" podID="7af4d913-b0bd-4af3-92e2-2ae1ad9debf4" containerID="e07012797319e730288b403cb408a47bdeea25667cf0d91db1fb306d6a3891f5" exitCode=0
Jul 18 16:46:13.793462 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:13.793417    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh" event={"ID":"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4","Type":"ContainerDied","Data":"e07012797319e730288b403cb408a47bdeea25667cf0d91db1fb306d6a3891f5"}
Jul 18 16:46:14.799257 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:14.799223    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh" event={"ID":"7af4d913-b0bd-4af3-92e2-2ae1ad9debf4","Type":"ContainerStarted","Data":"662015d3e56ac6cd13b8e3384ddd73e224ce61ab5174728455af8ddd4bd0c06f"}
Jul 18 16:46:14.799725 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:14.799353    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:14.821905 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:14.821832    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh" podStartSLOduration=2.290277309 podStartE2EDuration="6.82181422s" totalImagesPullingTime="3.516008964s" totalInitContainerRuntime="1.015527947s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:46:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:46:14.819994766 +0000 UTC m=+593.517825460" watchObservedRunningTime="2026-07-18 16:46:14.82181422 +0000 UTC m=+593.519644907"
Jul 18 16:46:19.624638 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:19.624605    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:20.050526 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:20.050443    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:20.150965 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:20.150930    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:20.314276 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:20.314201    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:20.414247 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:20.414215    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:23.527830 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:23.527797    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:24.682187 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:24.682136    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:25.133949 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:25.133917    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:25.234319 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:25.234288    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:25.804745 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:25.804715    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c768ddb5d-whpnh"
Jul 18 16:46:27.759774 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:27.759739    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:27.903233 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:27.903191    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:28.249602 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.249558    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:28.345004 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.344975    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5"]
Jul 18 16:46:28.350501 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.350481    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5"
Jul 18 16:46:28.354315 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.354292    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 18 16:46:28.354729 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.354709    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 18 16:46:28.354881 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.354763    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 18 16:46:28.355098 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.355076    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 18 16:46:28.355749 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.355735    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jwf5n\""
Jul 18 16:46:28.365696 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.365674    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5"]
Jul 18 16:46:28.487458 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.487416    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40d62415-b2ff-4eb9-9b5b-ec90c705184b-tmp\") pod \"kube-auth-proxy-6c475ff8c-hb9q5\" (UID: \"40d62415-b2ff-4eb9-9b5b-ec90c705184b\") " pod="openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5"
Jul 18 16:46:28.487458 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.487458    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d62415-b2ff-4eb9-9b5b-ec90c705184b-tls-certs\") pod \"kube-auth-proxy-6c475ff8c-hb9q5\" (UID: \"40d62415-b2ff-4eb9-9b5b-ec90c705184b\") " pod="openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5"
Jul 18 16:46:28.487703 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.487481    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhm5x\" (UniqueName: \"kubernetes.io/projected/40d62415-b2ff-4eb9-9b5b-ec90c705184b-kube-api-access-nhm5x\") pod \"kube-auth-proxy-6c475ff8c-hb9q5\" (UID: \"40d62415-b2ff-4eb9-9b5b-ec90c705184b\") " pod="openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5"
Jul 18 16:46:28.588701 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.588667    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40d62415-b2ff-4eb9-9b5b-ec90c705184b-tmp\") pod \"kube-auth-proxy-6c475ff8c-hb9q5\" (UID: \"40d62415-b2ff-4eb9-9b5b-ec90c705184b\") " pod="openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5"
Jul 18 16:46:28.588912 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.588713    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d62415-b2ff-4eb9-9b5b-ec90c705184b-tls-certs\") pod \"kube-auth-proxy-6c475ff8c-hb9q5\" (UID: \"40d62415-b2ff-4eb9-9b5b-ec90c705184b\") " pod="openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5"
Jul 18 16:46:28.588912 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.588735    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhm5x\" (UniqueName: \"kubernetes.io/projected/40d62415-b2ff-4eb9-9b5b-ec90c705184b-kube-api-access-nhm5x\") pod \"kube-auth-proxy-6c475ff8c-hb9q5\" (UID: \"40d62415-b2ff-4eb9-9b5b-ec90c705184b\") " pod="openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5"
Jul 18 16:46:28.591242 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.591212    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40d62415-b2ff-4eb9-9b5b-ec90c705184b-tmp\") pod \"kube-auth-proxy-6c475ff8c-hb9q5\" (UID: \"40d62415-b2ff-4eb9-9b5b-ec90c705184b\") " pod="openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5"
Jul 18 16:46:28.591518 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.591498    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40d62415-b2ff-4eb9-9b5b-ec90c705184b-tls-certs\") pod \"kube-auth-proxy-6c475ff8c-hb9q5\" (UID: \"40d62415-b2ff-4eb9-9b5b-ec90c705184b\") " pod="openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5"
Jul 18 16:46:28.619358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.619329    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhm5x\" (UniqueName: \"kubernetes.io/projected/40d62415-b2ff-4eb9-9b5b-ec90c705184b-kube-api-access-nhm5x\") pod \"kube-auth-proxy-6c475ff8c-hb9q5\" (UID: \"40d62415-b2ff-4eb9-9b5b-ec90c705184b\") " pod="openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5"
Jul 18 16:46:28.622407 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.622376    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:28.659918 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.659896    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5"
Jul 18 16:46:28.753525 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.753490    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:28.810093 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.810067    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5"]
Jul 18 16:46:28.812813 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:46:28.812778    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d62415_b2ff_4eb9_9b5b_ec90c705184b.slice/crio-257be6f2b19b5b1e5d129163ab2e7c2f45a7642b4115943e1472c102096a19f3 WatchSource:0}: Error finding container 257be6f2b19b5b1e5d129163ab2e7c2f45a7642b4115943e1472c102096a19f3: Status 404 returned error can't find the container with id 257be6f2b19b5b1e5d129163ab2e7c2f45a7642b4115943e1472c102096a19f3
Jul 18 16:46:28.814765 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.814748    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 16:46:28.851777 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.851696    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5" event={"ID":"40d62415-b2ff-4eb9-9b5b-ec90c705184b","Type":"ContainerStarted","Data":"257be6f2b19b5b1e5d129163ab2e7c2f45a7642b4115943e1472c102096a19f3"}
Jul 18 16:46:28.856224 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:28.856197    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:29.134375 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:29.134296    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:29.682303 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:29.682269    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:29.866918 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:29.866880    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:29.930077 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:29.930042    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:29.996643 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:29.996569    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:30.779630 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:30.779594    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:31.868533 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:31.868447    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5" event={"ID":"40d62415-b2ff-4eb9-9b5b-ec90c705184b","Type":"ContainerStarted","Data":"0ad858d2f9815a0e540665529a814642bd691ca90809be67bc7887ceb2e46955"}
Jul 18 16:46:31.885539 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:31.885494    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6c475ff8c-hb9q5" podStartSLOduration=1.09101522 podStartE2EDuration="3.885479437s" totalImagesPullingTime="2.794464217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:46:31.884084923 +0000 UTC m=+610.581915608" watchObservedRunningTime="2026-07-18 16:46:31.885479437 +0000 UTC m=+610.583310157"
Jul 18 16:46:34.139524 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.139486    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"]
Jul 18 16:46:34.142767 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.142750    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:34.146510 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.146491    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2njsn\""
Jul 18 16:46:34.146510 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.146498    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 18 16:46:34.146686 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.146497    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 18 16:46:34.146840 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.146824    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 18 16:46:34.146974 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.146957    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 18 16:46:34.147054 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.147038    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 18 16:46:34.152369 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.152347    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"]
Jul 18 16:46:34.242079 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.242035    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/038c1dca-874f-49b8-986c-cbe3e31c6721-metrics-cert\") pod \"lws-controller-manager-867f5b7897-mwfhn\" (UID: \"038c1dca-874f-49b8-986c-cbe3e31c6721\") " pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:34.242223 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.242107    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/038c1dca-874f-49b8-986c-cbe3e31c6721-cert\") pod \"lws-controller-manager-867f5b7897-mwfhn\" (UID: \"038c1dca-874f-49b8-986c-cbe3e31c6721\") " pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:34.242223 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.242140    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgf8b\" (UniqueName: \"kubernetes.io/projected/038c1dca-874f-49b8-986c-cbe3e31c6721-kube-api-access-tgf8b\") pod \"lws-controller-manager-867f5b7897-mwfhn\" (UID: \"038c1dca-874f-49b8-986c-cbe3e31c6721\") " pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:34.242223 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.242166    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/038c1dca-874f-49b8-986c-cbe3e31c6721-manager-config\") pod \"lws-controller-manager-867f5b7897-mwfhn\" (UID: \"038c1dca-874f-49b8-986c-cbe3e31c6721\") " pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:34.342862 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.342820    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/038c1dca-874f-49b8-986c-cbe3e31c6721-metrics-cert\") pod \"lws-controller-manager-867f5b7897-mwfhn\" (UID: \"038c1dca-874f-49b8-986c-cbe3e31c6721\") " pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:34.343037 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.342905    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/038c1dca-874f-49b8-986c-cbe3e31c6721-cert\") pod \"lws-controller-manager-867f5b7897-mwfhn\" (UID: \"038c1dca-874f-49b8-986c-cbe3e31c6721\") " pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:34.343037 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.342945    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgf8b\" (UniqueName: \"kubernetes.io/projected/038c1dca-874f-49b8-986c-cbe3e31c6721-kube-api-access-tgf8b\") pod \"lws-controller-manager-867f5b7897-mwfhn\" (UID: \"038c1dca-874f-49b8-986c-cbe3e31c6721\") " pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:34.343037 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.342978    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/038c1dca-874f-49b8-986c-cbe3e31c6721-manager-config\") pod \"lws-controller-manager-867f5b7897-mwfhn\" (UID: \"038c1dca-874f-49b8-986c-cbe3e31c6721\") " pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:34.343647 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.343624    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/038c1dca-874f-49b8-986c-cbe3e31c6721-manager-config\") pod \"lws-controller-manager-867f5b7897-mwfhn\" (UID: \"038c1dca-874f-49b8-986c-cbe3e31c6721\") " pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:34.345369 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.345350    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/038c1dca-874f-49b8-986c-cbe3e31c6721-cert\") pod \"lws-controller-manager-867f5b7897-mwfhn\" (UID: \"038c1dca-874f-49b8-986c-cbe3e31c6721\") " pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:34.345461 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.345413    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/038c1dca-874f-49b8-986c-cbe3e31c6721-metrics-cert\") pod \"lws-controller-manager-867f5b7897-mwfhn\" (UID: \"038c1dca-874f-49b8-986c-cbe3e31c6721\") " pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:34.352308 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.352273    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgf8b\" (UniqueName: \"kubernetes.io/projected/038c1dca-874f-49b8-986c-cbe3e31c6721-kube-api-access-tgf8b\") pod \"lws-controller-manager-867f5b7897-mwfhn\" (UID: \"038c1dca-874f-49b8-986c-cbe3e31c6721\") " pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:34.452540 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.452456    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:34.487821 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.487785    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:34.590150 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.590123    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"]
Jul 18 16:46:34.591978 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.591950    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:34.592189 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:46:34.592167    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038c1dca_874f_49b8_986c_cbe3e31c6721.slice/crio-f217c0b55e7f5e8aec1bed4b0040fe24e30f0fc2a5d9cfa379378703782b27ef WatchSource:0}: Error finding container f217c0b55e7f5e8aec1bed4b0040fe24e30f0fc2a5d9cfa379378703782b27ef: Status 404 returned error can't find the container with id f217c0b55e7f5e8aec1bed4b0040fe24e30f0fc2a5d9cfa379378703782b27ef
Jul 18 16:46:34.883254 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:34.883215    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn" event={"ID":"038c1dca-874f-49b8-986c-cbe3e31c6721","Type":"ContainerStarted","Data":"f217c0b55e7f5e8aec1bed4b0040fe24e30f0fc2a5d9cfa379378703782b27ef"}
Jul 18 16:46:35.180647 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:35.180416    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:35.589349 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:35.589318    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:35.796788 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:35.796755    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:37.898699 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:37.898652    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn" event={"ID":"038c1dca-874f-49b8-986c-cbe3e31c6721","Type":"ContainerStarted","Data":"dcdff9efa3d0a580e5671e9c596f181264cb6f9c79d2ad845f45d2911bc761ab"}
Jul 18 16:46:37.899095 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:37.898807    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:37.916703 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:37.916659    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn" podStartSLOduration=1.570570357 podStartE2EDuration="3.916644054s" totalImagesPullingTime="2.346073697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:46:37.914897048 +0000 UTC m=+616.612727732" watchObservedRunningTime="2026-07-18 16:46:37.916644054 +0000 UTC m=+616.614474738"
Jul 18 16:46:38.882692 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:38.882658    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:38.982605 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:38.982575    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:39.058537 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:39.058505    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:39.381577 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:39.381540    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:40.160833 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:40.160804    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:40.306571 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:40.306529    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:40.394433 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:40.394400    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:42.477747 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:42.477702    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:42.532549 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:42.532520    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:48.905834 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:48.905801    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-867f5b7897-mwfhn"
Jul 18 16:46:52.970178 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:52.970134    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:46:53.070127 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:46:53.070092    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:29.197342 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:29.197296    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:29.293054 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:29.293020    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:29.373489 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:29.373454    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:30.125457 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:30.125424    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:30.335825 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:30.335788    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:51.207617 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:51.207579    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:51.258403 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:51.258371    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:52.578342 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:52.578304    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:55.068257 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:55.068218    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:55.167675 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:55.167631    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:55.267341 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:55.267315    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:55.321730 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:55.321630    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:55.777667 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:55.777639    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:55.828829 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:55.828784    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:56.267538 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:56.267482    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:47:58.402463 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:47:58.402430    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:00.193966 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:00.193924    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:00.589512 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:00.589482    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:02.058544 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:02.058511    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:02.402066 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:02.402025    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9sqhw"]
Jul 18 16:48:02.404434 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:02.404416    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9sqhw"
Jul 18 16:48:02.406996 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:02.406978    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 18 16:48:02.407978 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:02.407962    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jfxqq\""
Jul 18 16:48:02.408063 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:02.408014    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 18 16:48:02.414324 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:02.414304    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9sqhw"]
Jul 18 16:48:02.482787 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:02.482751    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtz9w\" (UniqueName: \"kubernetes.io/projected/53346a72-5b96-4904-8537-e16d68b3d7f6-kube-api-access-jtz9w\") pod \"authorino-operator-75cc8d5cb7-9sqhw\" (UID: \"53346a72-5b96-4904-8537-e16d68b3d7f6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9sqhw"
Jul 18 16:48:02.584080 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:02.584030    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtz9w\" (UniqueName: \"kubernetes.io/projected/53346a72-5b96-4904-8537-e16d68b3d7f6-kube-api-access-jtz9w\") pod \"authorino-operator-75cc8d5cb7-9sqhw\" (UID: \"53346a72-5b96-4904-8537-e16d68b3d7f6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9sqhw"
Jul 18 16:48:02.592497 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:02.592466    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtz9w\" (UniqueName: \"kubernetes.io/projected/53346a72-5b96-4904-8537-e16d68b3d7f6-kube-api-access-jtz9w\") pod \"authorino-operator-75cc8d5cb7-9sqhw\" (UID: \"53346a72-5b96-4904-8537-e16d68b3d7f6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9sqhw"
Jul 18 16:48:02.715241 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:02.715159    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9sqhw"
Jul 18 16:48:02.844661 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:02.844640    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9sqhw"]
Jul 18 16:48:02.847183 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:48:02.847157    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53346a72_5b96_4904_8537_e16d68b3d7f6.slice/crio-8a7d6592bd56922c2ad6b82455eeb823e534e8ef39dd987fe329a4dff26823b2 WatchSource:0}: Error finding container 8a7d6592bd56922c2ad6b82455eeb823e534e8ef39dd987fe329a4dff26823b2: Status 404 returned error can't find the container with id 8a7d6592bd56922c2ad6b82455eeb823e534e8ef39dd987fe329a4dff26823b2
Jul 18 16:48:03.251648 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:03.251612    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9sqhw" event={"ID":"53346a72-5b96-4904-8537-e16d68b3d7f6","Type":"ContainerStarted","Data":"8a7d6592bd56922c2ad6b82455eeb823e534e8ef39dd987fe329a4dff26823b2"}
Jul 18 16:48:03.608089 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:03.608050    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:04.425696 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:04.425657    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:04.529256 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:04.529218    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:05.261670 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:05.261579    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9sqhw" event={"ID":"53346a72-5b96-4904-8537-e16d68b3d7f6","Type":"ContainerStarted","Data":"53956dc7c27d5b6b5b496cb4de6ace22163e905b28f2bf28a774c916259a72b1"}
Jul 18 16:48:05.261670 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:05.261667    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9sqhw"
Jul 18 16:48:05.277666 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:05.277621    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9sqhw" podStartSLOduration=1.26401649 podStartE2EDuration="3.277606623s" totalImagesPullingTime="2.013590133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:48:05.276804855 +0000 UTC m=+703.974635563" watchObservedRunningTime="2026-07-18 16:48:05.277606623 +0000 UTC m=+703.975437365"
Jul 18 16:48:05.700587 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:05.700556    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:05.808835 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:05.808801    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:05.871768 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:05.871735    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:06.337040 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:06.337013    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:06.435819 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:06.435789    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:08.392823 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:08.392777    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:08.452989 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:08.452956    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:09.329448 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:09.329418    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:15.818490 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:15.818409    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:15.919681 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:15.919637    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:16.268718 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:16.268687    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9sqhw"
Jul 18 16:48:16.489368 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:16.489334    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"]
Jul 18 16:48:16.491410 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:16.491393    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"
Jul 18 16:48:16.494012 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:16.493992    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bmv67\""
Jul 18 16:48:16.507887 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:16.507863    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"]
Jul 18 16:48:16.608923 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:16.608891    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b23c45b-cc4a-458f-b4d3-38c7504b2b72-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vqtts\" (UID: \"0b23c45b-cc4a-458f-b4d3-38c7504b2b72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"
Jul 18 16:48:16.609075 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:16.608993    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsgxz\" (UniqueName: \"kubernetes.io/projected/0b23c45b-cc4a-458f-b4d3-38c7504b2b72-kube-api-access-zsgxz\") pod \"kuadrant-operator-controller-manager-86878c769f-vqtts\" (UID: \"0b23c45b-cc4a-458f-b4d3-38c7504b2b72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"
Jul 18 16:48:16.709962 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:16.709935    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsgxz\" (UniqueName: \"kubernetes.io/projected/0b23c45b-cc4a-458f-b4d3-38c7504b2b72-kube-api-access-zsgxz\") pod \"kuadrant-operator-controller-manager-86878c769f-vqtts\" (UID: \"0b23c45b-cc4a-458f-b4d3-38c7504b2b72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"
Jul 18 16:48:16.710126 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:16.709983    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b23c45b-cc4a-458f-b4d3-38c7504b2b72-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vqtts\" (UID: \"0b23c45b-cc4a-458f-b4d3-38c7504b2b72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"
Jul 18 16:48:16.710351 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:16.710333    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b23c45b-cc4a-458f-b4d3-38c7504b2b72-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vqtts\" (UID: \"0b23c45b-cc4a-458f-b4d3-38c7504b2b72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"
Jul 18 16:48:16.721157 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:16.721135    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsgxz\" (UniqueName: \"kubernetes.io/projected/0b23c45b-cc4a-458f-b4d3-38c7504b2b72-kube-api-access-zsgxz\") pod \"kuadrant-operator-controller-manager-86878c769f-vqtts\" (UID: \"0b23c45b-cc4a-458f-b4d3-38c7504b2b72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"
Jul 18 16:48:16.801784 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:16.801749    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"
Jul 18 16:48:16.943750 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:16.943729    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"]
Jul 18 16:48:16.946340 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:48:16.946313    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b23c45b_cc4a_458f_b4d3_38c7504b2b72.slice/crio-93804ccac0824ae7d97199e03ced6c22aacfe09715cf63e41f196f2756676329 WatchSource:0}: Error finding container 93804ccac0824ae7d97199e03ced6c22aacfe09715cf63e41f196f2756676329: Status 404 returned error can't find the container with id 93804ccac0824ae7d97199e03ced6c22aacfe09715cf63e41f196f2756676329
Jul 18 16:48:17.314227 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:17.314132    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts" event={"ID":"0b23c45b-cc4a-458f-b4d3-38c7504b2b72","Type":"ContainerStarted","Data":"93804ccac0824ae7d97199e03ced6c22aacfe09715cf63e41f196f2756676329"}
Jul 18 16:48:20.612236 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:20.612195    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:20.709153 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:20.709112    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:21.334196 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:21.334160    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts" event={"ID":"0b23c45b-cc4a-458f-b4d3-38c7504b2b72","Type":"ContainerStarted","Data":"5644d66fb62e1ef006af6e90947a7aadf6245afa751dd0359c7a29f335a3f7bf"}
Jul 18 16:48:21.334353 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:21.334233    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"
Jul 18 16:48:21.353702 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:21.353652    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts" podStartSLOduration=1.364019225 podStartE2EDuration="5.353636025s" totalImagesPullingTime="3.9896168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:48:21.351283788 +0000 UTC m=+720.049114546" watchObservedRunningTime="2026-07-18 16:48:21.353636025 +0000 UTC m=+720.051466708"
Jul 18 16:48:23.037672 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:23.037639    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:23.138116 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:23.138085    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:28.600357 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:28.600326    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:28.701239 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:28.701209    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:32.340614 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:32.340584    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"
Jul 18 16:48:35.421157 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:35.421121    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:35.523184 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:35.523149    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:40.233751 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:40.233719    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:40.330099 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:40.330061    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:42.670027 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:42.669982    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:42.769774 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:42.769738    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:48.428905 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:48.428867    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:48.529400 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:48.529369    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:48.722064 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:48.721979    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:49.293224 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:49.293171    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:51.084942 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:51.084907    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:51.225033 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:51.224997    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:51.281876 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:51.281829    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:51.938917 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:51.938876    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:52.039585 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:52.039548    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:52.081955 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:52.081928    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-wrq4b"]
Jul 18 16:48:52.088564 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:52.088544    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wrq4b"
Jul 18 16:48:52.091104 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:52.091085    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qmp44\""
Jul 18 16:48:52.093605 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:52.093583    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wrq4b"]
Jul 18 16:48:52.168934 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:52.168903    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc5ql\" (UniqueName: \"kubernetes.io/projected/f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5-kube-api-access-sc5ql\") pod \"authorino-75d556bd77-wrq4b\" (UID: \"f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5\") " pod="kuadrant-system/authorino-75d556bd77-wrq4b"
Jul 18 16:48:52.270108 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:52.270024    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc5ql\" (UniqueName: \"kubernetes.io/projected/f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5-kube-api-access-sc5ql\") pod \"authorino-75d556bd77-wrq4b\" (UID: \"f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5\") " pod="kuadrant-system/authorino-75d556bd77-wrq4b"
Jul 18 16:48:52.278220 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:52.278196    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc5ql\" (UniqueName: \"kubernetes.io/projected/f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5-kube-api-access-sc5ql\") pod \"authorino-75d556bd77-wrq4b\" (UID: \"f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5\") " pod="kuadrant-system/authorino-75d556bd77-wrq4b"
Jul 18 16:48:52.405119 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:52.405085    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wrq4b"
Jul 18 16:48:52.534529 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:52.534374    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wrq4b"]
Jul 18 16:48:52.537182 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:48:52.537153    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4f3992c_5ad8_4475_9cd4_ef7da0ec70b5.slice/crio-785b3266ab0bdde5ab0983f9b81d1011c784c25fbe72974e1f24e91089ee7528 WatchSource:0}: Error finding container 785b3266ab0bdde5ab0983f9b81d1011c784c25fbe72974e1f24e91089ee7528: Status 404 returned error can't find the container with id 785b3266ab0bdde5ab0983f9b81d1011c784c25fbe72974e1f24e91089ee7528
Jul 18 16:48:53.485506 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:53.485458    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wrq4b" event={"ID":"f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5","Type":"ContainerStarted","Data":"785b3266ab0bdde5ab0983f9b81d1011c784c25fbe72974e1f24e91089ee7528"}
Jul 18 16:48:56.428435 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:56.428390    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:48:56.502282 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:56.502248    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wrq4b" event={"ID":"f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5","Type":"ContainerStarted","Data":"cfba7e4d8d9cedababc94d5c3d5bd5c451c1de1efb4e5a43236ff6163be6c478"}
Jul 18 16:48:56.523870 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:48:56.523806    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:01.027346 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:01.027307    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:01.128165 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:01.128121    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:01.228446 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:01.228415    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:01.287161 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:01.287062    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:04.029241 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:04.029203    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:04.085294 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:04.085262    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:07.031721 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:07.031675    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:07.127173 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:07.127138    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:07.636896 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:07.636855    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:20.315883 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:20.315832    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:20.416309 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:20.416276    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:22.059515 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.059449    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-wrq4b" podStartSLOduration=26.857651275 podStartE2EDuration="30.059424244s" totalImagesPullingTime="3.201772969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:48:56.518884912 +0000 UTC m=+755.216715597" watchObservedRunningTime="2026-07-18 16:49:22.059424244 +0000 UTC m=+780.757254929"
Jul 18 16:49:22.060591 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.060569    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wrq4b"]
Jul 18 16:49:22.060804 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.060781    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-wrq4b" podUID="f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5" containerName="authorino" containerID="cri-o://cfba7e4d8d9cedababc94d5c3d5bd5c451c1de1efb4e5a43236ff6163be6c478" gracePeriod=30
Jul 18 16:49:22.304217 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.304196    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wrq4b"
Jul 18 16:49:22.432415 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.432377    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc5ql\" (UniqueName: \"kubernetes.io/projected/f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5-kube-api-access-sc5ql\") pod \"f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5\" (UID: \"f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5\") "
Jul 18 16:49:22.434578 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.434559    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5-kube-api-access-sc5ql" (OuterVolumeSpecName: "kube-api-access-sc5ql") pod "f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5" (UID: "f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5"). InnerVolumeSpecName "kube-api-access-sc5ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 16:49:22.533121 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.533092    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc5ql\" (UniqueName: \"kubernetes.io/projected/f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5-kube-api-access-sc5ql\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:49:22.613664 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.613625    2554 generic.go:356] "Generic (PLEG): container finished" podID="f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5" containerID="cfba7e4d8d9cedababc94d5c3d5bd5c451c1de1efb4e5a43236ff6163be6c478" exitCode=0
Jul 18 16:49:22.613812 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.613674    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wrq4b"
Jul 18 16:49:22.613812 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.613704    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wrq4b" event={"ID":"f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5","Type":"ContainerDied","Data":"cfba7e4d8d9cedababc94d5c3d5bd5c451c1de1efb4e5a43236ff6163be6c478"}
Jul 18 16:49:22.613812 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.613747    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wrq4b" event={"ID":"f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5","Type":"ContainerDied","Data":"785b3266ab0bdde5ab0983f9b81d1011c784c25fbe72974e1f24e91089ee7528"}
Jul 18 16:49:22.613812 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.613764    2554 scope.go:117] "RemoveContainer" containerID="cfba7e4d8d9cedababc94d5c3d5bd5c451c1de1efb4e5a43236ff6163be6c478"
Jul 18 16:49:22.622612 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.622593    2554 scope.go:117] "RemoveContainer" containerID="cfba7e4d8d9cedababc94d5c3d5bd5c451c1de1efb4e5a43236ff6163be6c478"
Jul 18 16:49:22.622922 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:49:22.622896    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfba7e4d8d9cedababc94d5c3d5bd5c451c1de1efb4e5a43236ff6163be6c478\": container with ID starting with cfba7e4d8d9cedababc94d5c3d5bd5c451c1de1efb4e5a43236ff6163be6c478 not found: ID does not exist" containerID="cfba7e4d8d9cedababc94d5c3d5bd5c451c1de1efb4e5a43236ff6163be6c478"
Jul 18 16:49:22.623023 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.622933    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfba7e4d8d9cedababc94d5c3d5bd5c451c1de1efb4e5a43236ff6163be6c478"} err="failed to get container status \"cfba7e4d8d9cedababc94d5c3d5bd5c451c1de1efb4e5a43236ff6163be6c478\": rpc error: code = NotFound desc = could not find container \"cfba7e4d8d9cedababc94d5c3d5bd5c451c1de1efb4e5a43236ff6163be6c478\": container with ID starting with cfba7e4d8d9cedababc94d5c3d5bd5c451c1de1efb4e5a43236ff6163be6c478 not found: ID does not exist"
Jul 18 16:49:22.639545 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.637062    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wrq4b"]
Jul 18 16:49:22.643549 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:22.643522    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wrq4b"]
Jul 18 16:49:23.765663 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:23.765633    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5" path="/var/lib/kubelet/pods/f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5/volumes"
Jul 18 16:49:26.548864 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:26.548813    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:26.647792 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:26.647747    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:26.729224 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:26.729195    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:26.793674 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:26.793645    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:29.967895 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:29.967834    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:30.077640 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:30.077601    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:49:48.675619 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:49:48.675549    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:50:00.793236 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:00.793197    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:50:00.892167 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:00.892135    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:50:01.288261 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:01.288219    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:50:01.388507 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:01.388478    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:50:01.752108 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:01.752079    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:50:01.852340 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:01.852300    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:50:10.233703 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:10.233668    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:50:10.335332 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:10.335294    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:50:16.379564 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:16.379530    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:50:18.317361 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:18.317329    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:50:31.102981 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:31.102948    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:50:31.200593 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:31.200559    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:50:45.297393 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:45.297358    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:50:57.268872 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.268819    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"]
Jul 18 16:50:57.269326 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.269309    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5" containerName="authorino"
Jul 18 16:50:57.269374 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.269328    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5" containerName="authorino"
Jul 18 16:50:57.269415 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.269402    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4f3992c-5ad8-4475-9cd4-ef7da0ec70b5" containerName="authorino"
Jul 18 16:50:57.272466 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.272447    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.275926 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.275900    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 18 16:50:57.276051 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.275985    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 18 16:50:57.276051 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.275912    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 18 16:50:57.276161 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.275906    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9d62d\""
Jul 18 16:50:57.281539 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.281512    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"]
Jul 18 16:50:57.300067 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.300041    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:50:57.331191 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.331164    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f878d9-0b44-46e7-abea-3550442073ad-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.331296 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.331208    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f878d9-0b44-46e7-abea-3550442073ad-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.331296 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.331232    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxnb5\" (UniqueName: \"kubernetes.io/projected/46f878d9-0b44-46e7-abea-3550442073ad-kube-api-access-qxnb5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.331413 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.331341    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f878d9-0b44-46e7-abea-3550442073ad-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.331413 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.331398    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f878d9-0b44-46e7-abea-3550442073ad-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.331512 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.331447    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f878d9-0b44-46e7-abea-3550442073ad-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.432111 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.432075    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f878d9-0b44-46e7-abea-3550442073ad-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.432276 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.432126    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f878d9-0b44-46e7-abea-3550442073ad-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.432276 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.432151    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxnb5\" (UniqueName: \"kubernetes.io/projected/46f878d9-0b44-46e7-abea-3550442073ad-kube-api-access-qxnb5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.432276 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.432199    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f878d9-0b44-46e7-abea-3550442073ad-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.432276 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.432236    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f878d9-0b44-46e7-abea-3550442073ad-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.432276 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.432271    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f878d9-0b44-46e7-abea-3550442073ad-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.432539 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.432517    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f878d9-0b44-46e7-abea-3550442073ad-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.432611 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.432589    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f878d9-0b44-46e7-abea-3550442073ad-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.432672 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.432640    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f878d9-0b44-46e7-abea-3550442073ad-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.434523 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.434501    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f878d9-0b44-46e7-abea-3550442073ad-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.434884 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.434863    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f878d9-0b44-46e7-abea-3550442073ad-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.439630 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.439609    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxnb5\" (UniqueName: \"kubernetes.io/projected/46f878d9-0b44-46e7-abea-3550442073ad-kube-api-access-qxnb5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff\" (UID: \"46f878d9-0b44-46e7-abea-3550442073ad\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.583581 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.583549    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:50:57.715888 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:57.714053    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"]
Jul 18 16:50:58.011599 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:58.011486    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff" event={"ID":"46f878d9-0b44-46e7-abea-3550442073ad","Type":"ContainerStarted","Data":"01d864bebc75c7ac4c8841a6db2a421d89a220ce19fd69b0fb663787fb530ebd"}
Jul 18 16:50:59.017070 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:59.016973    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff" event={"ID":"46f878d9-0b44-46e7-abea-3550442073ad","Type":"ContainerStarted","Data":"fbd90df87d274a392d1d8a98a25688b071674cfc7931f87a4b4e536dc46cbada"}
Jul 18 16:50:59.035075 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:50:59.035028    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff" podStartSLOduration=1.023660705 podStartE2EDuration="2.035012356s" totalImagesPullingTime="1.011351651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:50:59.033801333 +0000 UTC m=+877.731632030" watchObservedRunningTime="2026-07-18 16:50:59.035012356 +0000 UTC m=+877.732843041"
Jul 18 16:51:07.583658 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:07.583625    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:51:07.584148 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:07.583686    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:51:07.596649 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:07.596620    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:51:08.072646 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:08.072571    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff"
Jul 18 16:51:08.480721 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:08.480687    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:51:12.560195 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:12.560155    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:51:12.649949 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:12.649912    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:51:29.268486 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.268456    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"]
Jul 18 16:51:29.272943 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.272924    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.275471 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.275450    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 18 16:51:29.281492 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.281471    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"]
Jul 18 16:51:29.304197 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.304164    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4x8k\" (UniqueName: \"kubernetes.io/projected/59d53068-56b7-4231-be50-ad6fa66a1f7d-kube-api-access-c4x8k\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.304343 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.304211    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59d53068-56b7-4231-be50-ad6fa66a1f7d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.304343 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.304238    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59d53068-56b7-4231-be50-ad6fa66a1f7d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.304343 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.304301    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59d53068-56b7-4231-be50-ad6fa66a1f7d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.304446 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.304345    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59d53068-56b7-4231-be50-ad6fa66a1f7d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.304446 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.304394    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59d53068-56b7-4231-be50-ad6fa66a1f7d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.319107 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.319076    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:51:29.404934 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.404891    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59d53068-56b7-4231-be50-ad6fa66a1f7d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.404934 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.404938    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59d53068-56b7-4231-be50-ad6fa66a1f7d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.405148 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.404966    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59d53068-56b7-4231-be50-ad6fa66a1f7d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.405148 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.404989    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59d53068-56b7-4231-be50-ad6fa66a1f7d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.405148 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.405039    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59d53068-56b7-4231-be50-ad6fa66a1f7d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.405148 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.405091    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4x8k\" (UniqueName: \"kubernetes.io/projected/59d53068-56b7-4231-be50-ad6fa66a1f7d-kube-api-access-c4x8k\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.405358 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.405326    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59d53068-56b7-4231-be50-ad6fa66a1f7d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.405428 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.405410    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59d53068-56b7-4231-be50-ad6fa66a1f7d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.405481 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.405451    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59d53068-56b7-4231-be50-ad6fa66a1f7d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.407199 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.407175    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59d53068-56b7-4231-be50-ad6fa66a1f7d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.407623 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.407605    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59d53068-56b7-4231-be50-ad6fa66a1f7d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.416711 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.416683    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4x8k\" (UniqueName: \"kubernetes.io/projected/59d53068-56b7-4231-be50-ad6fa66a1f7d-kube-api-access-c4x8k\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-z88hz\" (UID: \"59d53068-56b7-4231-be50-ad6fa66a1f7d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.592372 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.592346    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:29.724704 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.724680    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"]
Jul 18 16:51:29.726996 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:51:29.726965    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59d53068_56b7_4231_be50_ad6fa66a1f7d.slice/crio-1bd4a4be2812459528f38f5251d9c76aee974d2dc8b3e179551bdc10a6a0303a WatchSource:0}: Error finding container 1bd4a4be2812459528f38f5251d9c76aee974d2dc8b3e179551bdc10a6a0303a: Status 404 returned error can't find the container with id 1bd4a4be2812459528f38f5251d9c76aee974d2dc8b3e179551bdc10a6a0303a
Jul 18 16:51:29.728196 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:29.728180    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 16:51:30.159132 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:30.159031    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz" event={"ID":"59d53068-56b7-4231-be50-ad6fa66a1f7d","Type":"ContainerStarted","Data":"483c6cc060ac69d130d6976ca8b955f21e35eeaf447cfee8a2ec708120d7e28c"}
Jul 18 16:51:30.159132 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:30.159081    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz" event={"ID":"59d53068-56b7-4231-be50-ad6fa66a1f7d","Type":"ContainerStarted","Data":"1bd4a4be2812459528f38f5251d9c76aee974d2dc8b3e179551bdc10a6a0303a"}
Jul 18 16:51:30.178402 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:30.177618    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz" podStartSLOduration=1.009812662 podStartE2EDuration="1.177602205s" totalImagesPullingTime="167.789543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:51:30.176736122 +0000 UTC m=+908.874566806" watchObservedRunningTime="2026-07-18 16:51:30.177602205 +0000 UTC m=+908.875432888"
Jul 18 16:51:38.934137 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:38.934104    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:51:39.593021 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:39.592984    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:39.593182 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:39.593041    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:39.605481 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:39.605457    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:51:40.212714 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:51:40.212685    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-z88hz"
Jul 18 16:52:38.414801 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:38.414759    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d86c54866-njv76"]
Jul 18 16:52:38.418227 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:38.418206    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d86c54866-njv76"
Jul 18 16:52:38.421573 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:38.421555    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qmp44\""
Jul 18 16:52:38.421573 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:38.421564    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 18 16:52:38.426279 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:38.426257    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d86c54866-njv76"]
Jul 18 16:52:38.474942 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:38.474913    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvq55\" (UniqueName: \"kubernetes.io/projected/92cf2d68-2200-4256-a36b-0465b913a01f-kube-api-access-kvq55\") pod \"authorino-d86c54866-njv76\" (UID: \"92cf2d68-2200-4256-a36b-0465b913a01f\") " pod="kuadrant-system/authorino-d86c54866-njv76"
Jul 18 16:52:38.475104 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:38.474975    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/92cf2d68-2200-4256-a36b-0465b913a01f-tls-cert\") pod \"authorino-d86c54866-njv76\" (UID: \"92cf2d68-2200-4256-a36b-0465b913a01f\") " pod="kuadrant-system/authorino-d86c54866-njv76"
Jul 18 16:52:38.576136 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:38.576102    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/92cf2d68-2200-4256-a36b-0465b913a01f-tls-cert\") pod \"authorino-d86c54866-njv76\" (UID: \"92cf2d68-2200-4256-a36b-0465b913a01f\") " pod="kuadrant-system/authorino-d86c54866-njv76"
Jul 18 16:52:38.576291 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:38.576205    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvq55\" (UniqueName: \"kubernetes.io/projected/92cf2d68-2200-4256-a36b-0465b913a01f-kube-api-access-kvq55\") pod \"authorino-d86c54866-njv76\" (UID: \"92cf2d68-2200-4256-a36b-0465b913a01f\") " pod="kuadrant-system/authorino-d86c54866-njv76"
Jul 18 16:52:38.578880 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:38.578836    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/92cf2d68-2200-4256-a36b-0465b913a01f-tls-cert\") pod \"authorino-d86c54866-njv76\" (UID: \"92cf2d68-2200-4256-a36b-0465b913a01f\") " pod="kuadrant-system/authorino-d86c54866-njv76"
Jul 18 16:52:38.583638 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:38.583612    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvq55\" (UniqueName: \"kubernetes.io/projected/92cf2d68-2200-4256-a36b-0465b913a01f-kube-api-access-kvq55\") pod \"authorino-d86c54866-njv76\" (UID: \"92cf2d68-2200-4256-a36b-0465b913a01f\") " pod="kuadrant-system/authorino-d86c54866-njv76"
Jul 18 16:52:38.728178 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:38.728097    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d86c54866-njv76"
Jul 18 16:52:38.833368 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:38.833298    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:52:38.864605 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:38.864558    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d86c54866-njv76"]
Jul 18 16:52:38.867579 ip-10-0-142-77 kubenswrapper[2554]: W0718 16:52:38.867552    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92cf2d68_2200_4256_a36b_0465b913a01f.slice/crio-d7ecf4f424b1a210e11949bc10a35cd957cb836f56e65ed3ba44e962c5f3acb2 WatchSource:0}: Error finding container d7ecf4f424b1a210e11949bc10a35cd957cb836f56e65ed3ba44e962c5f3acb2: Status 404 returned error can't find the container with id d7ecf4f424b1a210e11949bc10a35cd957cb836f56e65ed3ba44e962c5f3acb2
Jul 18 16:52:38.924041 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:38.924006    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:52:39.454796 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:39.454764    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d86c54866-njv76" event={"ID":"92cf2d68-2200-4256-a36b-0465b913a01f","Type":"ContainerStarted","Data":"1d3125d829e215b500532ec1402fe27a08ca2fb7fde1457316fba0f2ebdfa560"}
Jul 18 16:52:39.454796 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:39.454801    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d86c54866-njv76" event={"ID":"92cf2d68-2200-4256-a36b-0465b913a01f","Type":"ContainerStarted","Data":"d7ecf4f424b1a210e11949bc10a35cd957cb836f56e65ed3ba44e962c5f3acb2"}
Jul 18 16:52:39.472573 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:39.472515    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d86c54866-njv76" podStartSLOduration=1.083646867 podStartE2EDuration="1.472493754s" totalImagesPullingTime="388.846887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 16:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 16:52:39.470188671 +0000 UTC m=+978.168019355" watchObservedRunningTime="2026-07-18 16:52:39.472493754 +0000 UTC m=+978.170324439"
Jul 18 16:52:39.919836 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:39.919796    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:52:40.025038 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:40.025007    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:52:50.051937 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:50.051836    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:52:50.106666 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:52:50.106632    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:54:46.331198 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:54:46.331109    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:54:47.552437 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:54:47.552402    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:55:05.437590 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:55:05.437556    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:55:05.489487 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:55:05.489459    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:55:17.716789 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:55:17.716748    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:55:17.818345 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:55:17.818313    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:55:17.919085 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:55:17.919045    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:55:23.634566 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:55:23.634531    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:55:23.689467 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:55:23.689435    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:55:23.768953 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:55:23.768914    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:55:23.868177 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:55:23.868143    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:55:23.967419 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:55:23.967334    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:55:29.913884 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:55:29.913832    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:55:30.085587 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:55:30.085554    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:55:33.110498 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:55:33.110461    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:55:33.160881 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:55:33.160818    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:57:57.731565 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:57:57.731530    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 16:59:25.993479 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:25.993443    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"]
Jul 18 16:59:25.994117 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:25.993657    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts" podUID="0b23c45b-cc4a-458f-b4d3-38c7504b2b72" containerName="manager" containerID="cri-o://5644d66fb62e1ef006af6e90947a7aadf6245afa751dd0359c7a29f335a3f7bf" gracePeriod=10
Jul 18 16:59:28.442543 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:28.442516    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"
Jul 18 16:59:28.489526 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:28.489497    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b23c45b-cc4a-458f-b4d3-38c7504b2b72-extensions-socket-volume\") pod \"0b23c45b-cc4a-458f-b4d3-38c7504b2b72\" (UID: \"0b23c45b-cc4a-458f-b4d3-38c7504b2b72\") "
Jul 18 16:59:28.489706 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:28.489544    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsgxz\" (UniqueName: \"kubernetes.io/projected/0b23c45b-cc4a-458f-b4d3-38c7504b2b72-kube-api-access-zsgxz\") pod \"0b23c45b-cc4a-458f-b4d3-38c7504b2b72\" (UID: \"0b23c45b-cc4a-458f-b4d3-38c7504b2b72\") "
Jul 18 16:59:28.489941 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:28.489917    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b23c45b-cc4a-458f-b4d3-38c7504b2b72-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0b23c45b-cc4a-458f-b4d3-38c7504b2b72" (UID: "0b23c45b-cc4a-458f-b4d3-38c7504b2b72"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 16:59:28.491746 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:28.491723    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b23c45b-cc4a-458f-b4d3-38c7504b2b72-kube-api-access-zsgxz" (OuterVolumeSpecName: "kube-api-access-zsgxz") pod "0b23c45b-cc4a-458f-b4d3-38c7504b2b72" (UID: "0b23c45b-cc4a-458f-b4d3-38c7504b2b72"). InnerVolumeSpecName "kube-api-access-zsgxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 16:59:28.590401 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:28.590368    2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b23c45b-cc4a-458f-b4d3-38c7504b2b72-extensions-socket-volume\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:59:28.590401 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:28.590398    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsgxz\" (UniqueName: \"kubernetes.io/projected/0b23c45b-cc4a-458f-b4d3-38c7504b2b72-kube-api-access-zsgxz\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 16:59:29.270171 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:29.270136    2554 generic.go:356] "Generic (PLEG): container finished" podID="0b23c45b-cc4a-458f-b4d3-38c7504b2b72" containerID="5644d66fb62e1ef006af6e90947a7aadf6245afa751dd0359c7a29f335a3f7bf" exitCode=0
Jul 18 16:59:29.270331 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:29.270184    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"
Jul 18 16:59:29.270331 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:29.270216    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts" event={"ID":"0b23c45b-cc4a-458f-b4d3-38c7504b2b72","Type":"ContainerDied","Data":"5644d66fb62e1ef006af6e90947a7aadf6245afa751dd0359c7a29f335a3f7bf"}
Jul 18 16:59:29.270331 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:29.270266    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts" event={"ID":"0b23c45b-cc4a-458f-b4d3-38c7504b2b72","Type":"ContainerDied","Data":"93804ccac0824ae7d97199e03ced6c22aacfe09715cf63e41f196f2756676329"}
Jul 18 16:59:29.270331 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:29.270286    2554 scope.go:117] "RemoveContainer" containerID="5644d66fb62e1ef006af6e90947a7aadf6245afa751dd0359c7a29f335a3f7bf"
Jul 18 16:59:29.279920 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:29.279896    2554 scope.go:117] "RemoveContainer" containerID="5644d66fb62e1ef006af6e90947a7aadf6245afa751dd0359c7a29f335a3f7bf"
Jul 18 16:59:29.280167 ip-10-0-142-77 kubenswrapper[2554]: E0718 16:59:29.280143    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5644d66fb62e1ef006af6e90947a7aadf6245afa751dd0359c7a29f335a3f7bf\": container with ID starting with 5644d66fb62e1ef006af6e90947a7aadf6245afa751dd0359c7a29f335a3f7bf not found: ID does not exist" containerID="5644d66fb62e1ef006af6e90947a7aadf6245afa751dd0359c7a29f335a3f7bf"
Jul 18 16:59:29.280218 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:29.280177    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5644d66fb62e1ef006af6e90947a7aadf6245afa751dd0359c7a29f335a3f7bf"} err="failed to get container status \"5644d66fb62e1ef006af6e90947a7aadf6245afa751dd0359c7a29f335a3f7bf\": rpc error: code = NotFound desc = could not find container \"5644d66fb62e1ef006af6e90947a7aadf6245afa751dd0359c7a29f335a3f7bf\": container with ID starting with 5644d66fb62e1ef006af6e90947a7aadf6245afa751dd0359c7a29f335a3f7bf not found: ID does not exist"
Jul 18 16:59:29.292468 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:29.292445    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"]
Jul 18 16:59:29.297509 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:29.297488    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vqtts"]
Jul 18 16:59:29.765440 ip-10-0-142-77 kubenswrapper[2554]: I0718 16:59:29.765409    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b23c45b-cc4a-458f-b4d3-38c7504b2b72" path="/var/lib/kubelet/pods/0b23c45b-cc4a-458f-b4d3-38c7504b2b72/volumes"
Jul 18 17:01:03.162751 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:01:03.162711    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:03:11.277199 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.277170    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"]
Jul 18 17:03:11.277658 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.277520    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b23c45b-cc4a-458f-b4d3-38c7504b2b72" containerName="manager"
Jul 18 17:03:11.277658 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.277530    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b23c45b-cc4a-458f-b4d3-38c7504b2b72" containerName="manager"
Jul 18 17:03:11.277658 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.277585    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b23c45b-cc4a-458f-b4d3-38c7504b2b72" containerName="manager"
Jul 18 17:03:11.280475 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.280459    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.282917 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.282894    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-4f6514dd-kserve-self-signed-certs\""
Jul 18 17:03:11.300054 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.300030    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"]
Jul 18 17:03:11.335453 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.335430    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:03:11.428281 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.428253    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-tls-certs\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.428402 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.428288    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-tmp-dir\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.428402 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.428318    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czdk9\" (UniqueName: \"kubernetes.io/projected/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-kube-api-access-czdk9\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.428402 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.428372    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-model-cache\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.428504 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.428409    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-home\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.428504 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.428436    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-dshm\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.529653 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.529574    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czdk9\" (UniqueName: \"kubernetes.io/projected/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-kube-api-access-czdk9\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.529653 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.529627    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-model-cache\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.529788 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.529674    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-home\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.529788 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.529708    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-dshm\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.529788 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.529745    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-tls-certs\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.529788 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.529772    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-tmp-dir\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.530120 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.530101    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-home\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.530205 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.530139    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-model-cache\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.530300 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.530279    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-tmp-dir\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.532118 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.532095    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-dshm\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.532412 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.532396    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-tls-certs\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.537907 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.537891    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czdk9\" (UniqueName: \"kubernetes.io/projected/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-kube-api-access-czdk9\") pod \"e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") " pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.589565 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.589534    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:11.717627 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:11.717555    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"]
Jul 18 17:03:11.720015 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:03:11.719989    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb95c2e5_a332_4680_9132_f2dcf7d0f92a.slice/crio-804fe6f56cee57fd140e4ed82491fb7059fab4ffc707c1e2af335586e50a7731 WatchSource:0}: Error finding container 804fe6f56cee57fd140e4ed82491fb7059fab4ffc707c1e2af335586e50a7731: Status 404 returned error can't find the container with id 804fe6f56cee57fd140e4ed82491fb7059fab4ffc707c1e2af335586e50a7731
Jul 18 17:03:12.229033 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:12.228997    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759" event={"ID":"bb95c2e5-a332-4680-9132-f2dcf7d0f92a","Type":"ContainerStarted","Data":"27a6366619f19da187c44b3035948a713a2559de1e5119053cbd882c24886a25"}
Jul 18 17:03:12.229198 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:12.229044    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759" event={"ID":"bb95c2e5-a332-4680-9132-f2dcf7d0f92a","Type":"ContainerStarted","Data":"804fe6f56cee57fd140e4ed82491fb7059fab4ffc707c1e2af335586e50a7731"}
Jul 18 17:03:12.248476 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:12.248429    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759" podStartSLOduration=1.248413669 podStartE2EDuration="1.248413669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:03:12.246065435 +0000 UTC m=+1610.943896119" watchObservedRunningTime="2026-07-18 17:03:12.248413669 +0000 UTC m=+1610.946244352"
Jul 18 17:03:12.887373 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:12.887300    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:03:21.371675 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.371644    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"]
Jul 18 17:03:21.374694 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.374672    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.377073 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.377049    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-4f6514dd-kserve-self-signed-certs\""
Jul 18 17:03:21.385182 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.385164    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"]
Jul 18 17:03:21.406040 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.406011    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:03:21.521519 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.521442    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fth4d\" (UniqueName: \"kubernetes.io/projected/7be0111f-a818-40be-8476-1ed1e99f17d7-kube-api-access-fth4d\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.521519 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.521488    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-dshm\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.521519 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.521512    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-model-cache\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.521719 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.521586    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-tmp-dir\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.521719 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.521636    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be0111f-a818-40be-8476-1ed1e99f17d7-tls-certs\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.521719 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.521690    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-home\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.589951 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.589920    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:21.590081 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.589963    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:21.602337 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.602317    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:21.622307 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.622275    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-dshm\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.622445 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.622311    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-model-cache\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.622445 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.622332    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-tmp-dir\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.622445 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.622356    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be0111f-a818-40be-8476-1ed1e99f17d7-tls-certs\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.622445 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.622410    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-home\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.622648 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.622499    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fth4d\" (UniqueName: \"kubernetes.io/projected/7be0111f-a818-40be-8476-1ed1e99f17d7-kube-api-access-fth4d\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.622720 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.622692    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-model-cache\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.622774 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.622736    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-tmp-dir\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.622820 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.622805    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-home\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.624909 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.624891    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-dshm\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.624992 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.624978    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be0111f-a818-40be-8476-1ed1e99f17d7-tls-certs\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.629969 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.629952    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fth4d\" (UniqueName: \"kubernetes.io/projected/7be0111f-a818-40be-8476-1ed1e99f17d7-kube-api-access-fth4d\") pod \"e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") " pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.684609 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.684579    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:21.820378 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:21.820306    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"]
Jul 18 17:03:21.823590 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:03:21.823562    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7be0111f_a818_40be_8476_1ed1e99f17d7.slice/crio-888d9a63fb05193dbbbfe56f30cdbd8df9bdbd3df685f4c46453342bd2b528fe WatchSource:0}: Error finding container 888d9a63fb05193dbbbfe56f30cdbd8df9bdbd3df685f4c46453342bd2b528fe: Status 404 returned error can't find the container with id 888d9a63fb05193dbbbfe56f30cdbd8df9bdbd3df685f4c46453342bd2b528fe
Jul 18 17:03:22.273676 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:22.273635    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5" event={"ID":"7be0111f-a818-40be-8476-1ed1e99f17d7","Type":"ContainerStarted","Data":"8d4fd80b29d16800b683a67446bf3411b32345816232d76eec5c2f9e3af9c801"}
Jul 18 17:03:22.273894 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:22.273684    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5" event={"ID":"7be0111f-a818-40be-8476-1ed1e99f17d7","Type":"ContainerStarted","Data":"888d9a63fb05193dbbbfe56f30cdbd8df9bdbd3df685f4c46453342bd2b528fe"}
Jul 18 17:03:22.284815 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:22.284791    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:03:22.290784 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:22.290746    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5" podStartSLOduration=1.290732073 podStartE2EDuration="1.290732073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:03:22.289883796 +0000 UTC m=+1620.987714481" watchObservedRunningTime="2026-07-18 17:03:22.290732073 +0000 UTC m=+1620.988562756"
Jul 18 17:03:31.685569 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:31.685533    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:31.685569 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:31.685577    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:31.698333 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:31.698310    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:32.331180 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:32.331147    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:03:36.553957 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:36.553922    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:03:36.609810 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:36.609776    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:03:57.954150 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:57.954098    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"]
Jul 18 17:03:57.973695 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:57.973666    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:03:58.983622 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:58.983589    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"]
Jul 18 17:03:58.993409 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:03:58.993375    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:04:12.974593 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:12.974537    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759" podUID="bb95c2e5-a332-4680-9132-f2dcf7d0f92a" containerName="main" containerID="cri-o://27a6366619f19da187c44b3035948a713a2559de1e5119053cbd882c24886a25" gracePeriod=45
Jul 18 17:04:13.220231 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.220208    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:04:13.282286 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.282190    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-tmp-dir\") pod \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") "
Jul 18 17:04:13.282286 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.282230    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-home\") pod \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") "
Jul 18 17:04:13.282286 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.282287    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-dshm\") pod \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") "
Jul 18 17:04:13.282561 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.282323    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-tls-certs\") pod \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") "
Jul 18 17:04:13.282561 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.282376    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czdk9\" (UniqueName: \"kubernetes.io/projected/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-kube-api-access-czdk9\") pod \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") "
Jul 18 17:04:13.282561 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.282403    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-model-cache\") pod \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\" (UID: \"bb95c2e5-a332-4680-9132-f2dcf7d0f92a\") "
Jul 18 17:04:13.282561 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.282495    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb95c2e5-a332-4680-9132-f2dcf7d0f92a" (UID: "bb95c2e5-a332-4680-9132-f2dcf7d0f92a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:04:13.282561 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.282506    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-home" (OuterVolumeSpecName: "home") pod "bb95c2e5-a332-4680-9132-f2dcf7d0f92a" (UID: "bb95c2e5-a332-4680-9132-f2dcf7d0f92a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:04:13.282804 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.282706    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-tmp-dir\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:04:13.282804 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.282719    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-home\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:04:13.282804 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.282726    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb95c2e5-a332-4680-9132-f2dcf7d0f92a" (UID: "bb95c2e5-a332-4680-9132-f2dcf7d0f92a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:04:13.284730 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.284705    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-dshm" (OuterVolumeSpecName: "dshm") pod "bb95c2e5-a332-4680-9132-f2dcf7d0f92a" (UID: "bb95c2e5-a332-4680-9132-f2dcf7d0f92a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:04:13.284799 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.284732    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-kube-api-access-czdk9" (OuterVolumeSpecName: "kube-api-access-czdk9") pod "bb95c2e5-a332-4680-9132-f2dcf7d0f92a" (UID: "bb95c2e5-a332-4680-9132-f2dcf7d0f92a"). InnerVolumeSpecName "kube-api-access-czdk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:04:13.284799 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.284786    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb95c2e5-a332-4680-9132-f2dcf7d0f92a" (UID: "bb95c2e5-a332-4680-9132-f2dcf7d0f92a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:04:13.383389 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.383353    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-dshm\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:04:13.383389 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.383382    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-tls-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:04:13.383389 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.383394    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czdk9\" (UniqueName: \"kubernetes.io/projected/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-kube-api-access-czdk9\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:04:13.383619 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.383403    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb95c2e5-a332-4680-9132-f2dcf7d0f92a-model-cache\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:04:13.496518 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.496480    2554 generic.go:356] "Generic (PLEG): container finished" podID="bb95c2e5-a332-4680-9132-f2dcf7d0f92a" containerID="27a6366619f19da187c44b3035948a713a2559de1e5119053cbd882c24886a25" exitCode=0
Jul 18 17:04:13.496674 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.496527    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"
Jul 18 17:04:13.496674 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.496568    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759" event={"ID":"bb95c2e5-a332-4680-9132-f2dcf7d0f92a","Type":"ContainerDied","Data":"27a6366619f19da187c44b3035948a713a2559de1e5119053cbd882c24886a25"}
Jul 18 17:04:13.496674 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.496613    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759" event={"ID":"bb95c2e5-a332-4680-9132-f2dcf7d0f92a","Type":"ContainerDied","Data":"804fe6f56cee57fd140e4ed82491fb7059fab4ffc707c1e2af335586e50a7731"}
Jul 18 17:04:13.496674 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.496632    2554 scope.go:117] "RemoveContainer" containerID="27a6366619f19da187c44b3035948a713a2559de1e5119053cbd882c24886a25"
Jul 18 17:04:13.505252 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.505236    2554 scope.go:117] "RemoveContainer" containerID="27a6366619f19da187c44b3035948a713a2559de1e5119053cbd882c24886a25"
Jul 18 17:04:13.505476 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:04:13.505461    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a6366619f19da187c44b3035948a713a2559de1e5119053cbd882c24886a25\": container with ID starting with 27a6366619f19da187c44b3035948a713a2559de1e5119053cbd882c24886a25 not found: ID does not exist" containerID="27a6366619f19da187c44b3035948a713a2559de1e5119053cbd882c24886a25"
Jul 18 17:04:13.505521 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.505486    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a6366619f19da187c44b3035948a713a2559de1e5119053cbd882c24886a25"} err="failed to get container status \"27a6366619f19da187c44b3035948a713a2559de1e5119053cbd882c24886a25\": rpc error: code = NotFound desc = could not find container \"27a6366619f19da187c44b3035948a713a2559de1e5119053cbd882c24886a25\": container with ID starting with 27a6366619f19da187c44b3035948a713a2559de1e5119053cbd882c24886a25 not found: ID does not exist"
Jul 18 17:04:13.516208 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.516183    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"]
Jul 18 17:04:13.520581 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.520563    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-4f6514dd-kserve-5f45d597cf-s4759"]
Jul 18 17:04:13.765719 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:13.765687    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb95c2e5-a332-4680-9132-f2dcf7d0f92a" path="/var/lib/kubelet/pods/bb95c2e5-a332-4680-9132-f2dcf7d0f92a/volumes"
Jul 18 17:04:14.003820 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.003772    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5" podUID="7be0111f-a818-40be-8476-1ed1e99f17d7" containerName="main" containerID="cri-o://8d4fd80b29d16800b683a67446bf3411b32345816232d76eec5c2f9e3af9c801" gracePeriod=45
Jul 18 17:04:14.249962 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.249935    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:04:14.292418 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.292333    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-model-cache\") pod \"7be0111f-a818-40be-8476-1ed1e99f17d7\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") "
Jul 18 17:04:14.292418 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.292367    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fth4d\" (UniqueName: \"kubernetes.io/projected/7be0111f-a818-40be-8476-1ed1e99f17d7-kube-api-access-fth4d\") pod \"7be0111f-a818-40be-8476-1ed1e99f17d7\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") "
Jul 18 17:04:14.292418 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.292383    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-dshm\") pod \"7be0111f-a818-40be-8476-1ed1e99f17d7\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") "
Jul 18 17:04:14.292418 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.292405    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be0111f-a818-40be-8476-1ed1e99f17d7-tls-certs\") pod \"7be0111f-a818-40be-8476-1ed1e99f17d7\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") "
Jul 18 17:04:14.292717 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.292433    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-tmp-dir\") pod \"7be0111f-a818-40be-8476-1ed1e99f17d7\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") "
Jul 18 17:04:14.292717 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.292475    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-home\") pod \"7be0111f-a818-40be-8476-1ed1e99f17d7\" (UID: \"7be0111f-a818-40be-8476-1ed1e99f17d7\") "
Jul 18 17:04:14.292717 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.292673    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "7be0111f-a818-40be-8476-1ed1e99f17d7" (UID: "7be0111f-a818-40be-8476-1ed1e99f17d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:04:14.292717 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.292696    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7be0111f-a818-40be-8476-1ed1e99f17d7" (UID: "7be0111f-a818-40be-8476-1ed1e99f17d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:04:14.292944 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.292799    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-home" (OuterVolumeSpecName: "home") pod "7be0111f-a818-40be-8476-1ed1e99f17d7" (UID: "7be0111f-a818-40be-8476-1ed1e99f17d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:04:14.294703 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.294676    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-dshm" (OuterVolumeSpecName: "dshm") pod "7be0111f-a818-40be-8476-1ed1e99f17d7" (UID: "7be0111f-a818-40be-8476-1ed1e99f17d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:04:14.294703 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.294690    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be0111f-a818-40be-8476-1ed1e99f17d7-kube-api-access-fth4d" (OuterVolumeSpecName: "kube-api-access-fth4d") pod "7be0111f-a818-40be-8476-1ed1e99f17d7" (UID: "7be0111f-a818-40be-8476-1ed1e99f17d7"). InnerVolumeSpecName "kube-api-access-fth4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:04:14.294703 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.294686    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7be0111f-a818-40be-8476-1ed1e99f17d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7be0111f-a818-40be-8476-1ed1e99f17d7" (UID: "7be0111f-a818-40be-8476-1ed1e99f17d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:04:14.393553 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.393520    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-model-cache\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:04:14.393553 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.393547    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fth4d\" (UniqueName: \"kubernetes.io/projected/7be0111f-a818-40be-8476-1ed1e99f17d7-kube-api-access-fth4d\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:04:14.393553 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.393558    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-dshm\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:04:14.393767 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.393567    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be0111f-a818-40be-8476-1ed1e99f17d7-tls-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:04:14.393767 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.393575    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-tmp-dir\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:04:14.393767 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.393583    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7be0111f-a818-40be-8476-1ed1e99f17d7-home\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:04:14.504591 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.504555    2554 generic.go:356] "Generic (PLEG): container finished" podID="7be0111f-a818-40be-8476-1ed1e99f17d7" containerID="8d4fd80b29d16800b683a67446bf3411b32345816232d76eec5c2f9e3af9c801" exitCode=0
Jul 18 17:04:14.504753 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.504604    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"
Jul 18 17:04:14.504753 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.504610    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5" event={"ID":"7be0111f-a818-40be-8476-1ed1e99f17d7","Type":"ContainerDied","Data":"8d4fd80b29d16800b683a67446bf3411b32345816232d76eec5c2f9e3af9c801"}
Jul 18 17:04:14.504753 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.504706    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5" event={"ID":"7be0111f-a818-40be-8476-1ed1e99f17d7","Type":"ContainerDied","Data":"888d9a63fb05193dbbbfe56f30cdbd8df9bdbd3df685f4c46453342bd2b528fe"}
Jul 18 17:04:14.504753 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.504723    2554 scope.go:117] "RemoveContainer" containerID="8d4fd80b29d16800b683a67446bf3411b32345816232d76eec5c2f9e3af9c801"
Jul 18 17:04:14.513680 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.513663    2554 scope.go:117] "RemoveContainer" containerID="8d4fd80b29d16800b683a67446bf3411b32345816232d76eec5c2f9e3af9c801"
Jul 18 17:04:14.513961 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:04:14.513938    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4fd80b29d16800b683a67446bf3411b32345816232d76eec5c2f9e3af9c801\": container with ID starting with 8d4fd80b29d16800b683a67446bf3411b32345816232d76eec5c2f9e3af9c801 not found: ID does not exist" containerID="8d4fd80b29d16800b683a67446bf3411b32345816232d76eec5c2f9e3af9c801"
Jul 18 17:04:14.514021 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.513970    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4fd80b29d16800b683a67446bf3411b32345816232d76eec5c2f9e3af9c801"} err="failed to get container status \"8d4fd80b29d16800b683a67446bf3411b32345816232d76eec5c2f9e3af9c801\": rpc error: code = NotFound desc = could not find container \"8d4fd80b29d16800b683a67446bf3411b32345816232d76eec5c2f9e3af9c801\": container with ID starting with 8d4fd80b29d16800b683a67446bf3411b32345816232d76eec5c2f9e3af9c801 not found: ID does not exist"
Jul 18 17:04:14.524524 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.524503    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"]
Jul 18 17:04:14.527616 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:14.527595    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-4f6514dd-kserve-8765479d7-rldm5"]
Jul 18 17:04:15.765589 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:04:15.765558    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7be0111f-a818-40be-8476-1ed1e99f17d7" path="/var/lib/kubelet/pods/7be0111f-a818-40be-8476-1ed1e99f17d7/volumes"
Jul 18 17:05:15.415276 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:05:15.415240    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:05:49.513738 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:05:49.513695    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:08.507021 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:08.506985    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:16.009204 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:16.009162    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:24.267772 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:24.267726    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:29.380658 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:29.380620    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:32.311020 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:32.310978    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:32.411232 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:32.411200    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:32.742078 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:32.742046    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:38.741176 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:38.741139    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:38.795913 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:38.795879    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:39.047403 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:39.047319    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:39.275706 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:39.275677    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:39.387593 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:39.387560    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:41.635261 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.635224    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:41.731867 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.731810    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:41.859700 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.859672    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"]
Jul 18 17:07:41.860013 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.859989    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb95c2e5-a332-4680-9132-f2dcf7d0f92a" containerName="main"
Jul 18 17:07:41.860013 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.860005    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb95c2e5-a332-4680-9132-f2dcf7d0f92a" containerName="main"
Jul 18 17:07:41.860013 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.860021    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7be0111f-a818-40be-8476-1ed1e99f17d7" containerName="main"
Jul 18 17:07:41.860188 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.860026    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7be0111f-a818-40be-8476-1ed1e99f17d7" containerName="main"
Jul 18 17:07:41.860188 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.860087    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7be0111f-a818-40be-8476-1ed1e99f17d7" containerName="main"
Jul 18 17:07:41.860188 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.860096    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb95c2e5-a332-4680-9132-f2dcf7d0f92a" containerName="main"
Jul 18 17:07:41.862659 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.862640    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:07:41.866362 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.866341    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e2278917-serving-cert\""
Jul 18 17:07:41.866482 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.866356    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 18 17:07:41.866482 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.866347    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2pqsv\""
Jul 18 17:07:41.867383 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.867367    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 18 17:07:41.874763 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.874740    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"]
Jul 18 17:07:41.898203 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:41.898139    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:42.045700 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.045665    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-openshift-service-ca\") pod \"maas-api-e2e-ait-e2278917-859785f4bf-hw4zr\" (UID: \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:07:42.045700 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.045703    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvlgp\" (UniqueName: \"kubernetes.io/projected/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-kube-api-access-bvlgp\") pod \"maas-api-e2e-ait-e2278917-859785f4bf-hw4zr\" (UID: \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:07:42.045961 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.045737    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-maas-api-tls\") pod \"maas-api-e2e-ait-e2278917-859785f4bf-hw4zr\" (UID: \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:07:42.146789 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.146749    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-openshift-service-ca\") pod \"maas-api-e2e-ait-e2278917-859785f4bf-hw4zr\" (UID: \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:07:42.147006 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.146789    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvlgp\" (UniqueName: \"kubernetes.io/projected/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-kube-api-access-bvlgp\") pod \"maas-api-e2e-ait-e2278917-859785f4bf-hw4zr\" (UID: \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:07:42.147006 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.146840    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-maas-api-tls\") pod \"maas-api-e2e-ait-e2278917-859785f4bf-hw4zr\" (UID: \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:07:42.147006 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:07:42.146986    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-serving-cert: secret "maas-api-e2e-ait-e2278917-serving-cert" not found
Jul 18 17:07:42.147178 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:07:42.147039    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-maas-api-tls podName:ad8dc5a7-f33a-4c45-b5c3-389241d82d5e nodeName:}" failed. No retries permitted until 2026-07-18 17:07:42.64702369 +0000 UTC m=+1881.344854352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-maas-api-tls") pod "maas-api-e2e-ait-e2278917-859785f4bf-hw4zr" (UID: "ad8dc5a7-f33a-4c45-b5c3-389241d82d5e") : secret "maas-api-e2e-ait-e2278917-serving-cert" not found
Jul 18 17:07:42.147410 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.147390    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-openshift-service-ca\") pod \"maas-api-e2e-ait-e2278917-859785f4bf-hw4zr\" (UID: \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:07:42.155419 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.155368    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvlgp\" (UniqueName: \"kubernetes.io/projected/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-kube-api-access-bvlgp\") pod \"maas-api-e2e-ait-e2278917-859785f4bf-hw4zr\" (UID: \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:07:42.230827 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.230790    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:42.384090 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.384043    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:42.485046 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.484944    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:07:42.651813 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.651777    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-maas-api-tls\") pod \"maas-api-e2e-ait-e2278917-859785f4bf-hw4zr\" (UID: \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:07:42.654415 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.654396    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-maas-api-tls\") pod \"maas-api-e2e-ait-e2278917-859785f4bf-hw4zr\" (UID: \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:07:42.773137 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.773026    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:07:42.915677 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.915651    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"]
Jul 18 17:07:42.917760 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:07:42.917722    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8dc5a7_f33a_4c45_b5c3_389241d82d5e.slice/crio-1161d305ee77c739b973a85ff1f7876a68d165afcf4f6c335f9281034cbafcbc WatchSource:0}: Error finding container 1161d305ee77c739b973a85ff1f7876a68d165afcf4f6c335f9281034cbafcbc: Status 404 returned error can't find the container with id 1161d305ee77c739b973a85ff1f7876a68d165afcf4f6c335f9281034cbafcbc
Jul 18 17:07:42.919167 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:42.919151    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 17:07:43.400985 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:43.400954    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr" event={"ID":"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e","Type":"ContainerStarted","Data":"1161d305ee77c739b973a85ff1f7876a68d165afcf4f6c335f9281034cbafcbc"}
Jul 18 17:07:45.411395 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:45.411353    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr" event={"ID":"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e","Type":"ContainerStarted","Data":"b166cd30bea995edb16f5baec79c54bf8a533ba71167bc33c0e98da419892291"}
Jul 18 17:07:45.411876 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:45.411486    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:07:45.455192 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:45.455148    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr" podStartSLOduration=2.1010579050000002 podStartE2EDuration="4.455131758s" totalImagesPullingTime="2.354073853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:07:45.454793028 +0000 UTC m=+1884.152623711" watchObservedRunningTime="2026-07-18 17:07:45.455131758 +0000 UTC m=+1884.152962491"
Jul 18 17:07:51.421353 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:07:51.421325    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:08:07.129591 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:07.129558    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2278917-7grnx"]
Jul 18 17:08:07.133113 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:07.133097    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2278917-7grnx"
Jul 18 17:08:07.136473 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:07.136449    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7rq8\""
Jul 18 17:08:07.140857 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:07.140799    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2278917-7grnx"]
Jul 18 17:08:07.265481 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:07.265442    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbcf1bc1-0848-4782-8d00-41fd8d483ad8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2278917-7grnx\" (UID: \"fbcf1bc1-0848-4782-8d00-41fd8d483ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2278917-7grnx"
Jul 18 17:08:07.366751 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:07.366714    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbcf1bc1-0848-4782-8d00-41fd8d483ad8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2278917-7grnx\" (UID: \"fbcf1bc1-0848-4782-8d00-41fd8d483ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2278917-7grnx"
Jul 18 17:08:07.367367 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:07.367346    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbcf1bc1-0848-4782-8d00-41fd8d483ad8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2278917-7grnx\" (UID: \"fbcf1bc1-0848-4782-8d00-41fd8d483ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2278917-7grnx"
Jul 18 17:08:07.444883 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:07.444777    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2278917-7grnx"
Jul 18 17:08:07.574480 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:07.574456    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2278917-7grnx"]
Jul 18 17:08:07.576538 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:08:07.576506    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbcf1bc1_0848_4782_8d00_41fd8d483ad8.slice/crio-2bd395ebc5004f4459163b3bcbff3be215129cf99aa5c2a7525b198437050124 WatchSource:0}: Error finding container 2bd395ebc5004f4459163b3bcbff3be215129cf99aa5c2a7525b198437050124: Status 404 returned error can't find the container with id 2bd395ebc5004f4459163b3bcbff3be215129cf99aa5c2a7525b198437050124
Jul 18 17:08:08.511130 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:08.511094    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2278917-7grnx" event={"ID":"fbcf1bc1-0848-4782-8d00-41fd8d483ad8","Type":"ContainerStarted","Data":"2bd395ebc5004f4459163b3bcbff3be215129cf99aa5c2a7525b198437050124"}
Jul 18 17:08:10.521336 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:10.521254    2554 generic.go:356] "Generic (PLEG): container finished" podID="fbcf1bc1-0848-4782-8d00-41fd8d483ad8" containerID="4ecb5a189829873dd1cd04c152f990641a05f0d6fe9d5cd4cc289576ad7029c7" exitCode=0
Jul 18 17:08:10.521336 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:10.521300    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2278917-7grnx" event={"ID":"fbcf1bc1-0848-4782-8d00-41fd8d483ad8","Type":"ContainerDied","Data":"4ecb5a189829873dd1cd04c152f990641a05f0d6fe9d5cd4cc289576ad7029c7"}
Jul 18 17:08:11.657693 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:11.657669    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2278917-7grnx"
Jul 18 17:08:11.802781 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:11.802686    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbcf1bc1-0848-4782-8d00-41fd8d483ad8-maas-api-service-ca\") pod \"fbcf1bc1-0848-4782-8d00-41fd8d483ad8\" (UID: \"fbcf1bc1-0848-4782-8d00-41fd8d483ad8\") "
Jul 18 17:08:11.803051 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:11.803027    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbcf1bc1-0848-4782-8d00-41fd8d483ad8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fbcf1bc1-0848-4782-8d00-41fd8d483ad8" (UID: "fbcf1bc1-0848-4782-8d00-41fd8d483ad8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:08:11.903909 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:11.903876    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbcf1bc1-0848-4782-8d00-41fd8d483ad8-maas-api-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:08:12.531192 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:12.531162    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2278917-7grnx"
Jul 18 17:08:12.531192 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:12.531174    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2278917-7grnx" event={"ID":"fbcf1bc1-0848-4782-8d00-41fd8d483ad8","Type":"ContainerDied","Data":"2bd395ebc5004f4459163b3bcbff3be215129cf99aa5c2a7525b198437050124"}
Jul 18 17:08:12.531400 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:12.531219    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bd395ebc5004f4459163b3bcbff3be215129cf99aa5c2a7525b198437050124"
Jul 18 17:08:17.502959 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:17.502917    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"]
Jul 18 17:08:17.503406 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:17.503192    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr" podUID="ad8dc5a7-f33a-4c45-b5c3-389241d82d5e" containerName="maas-api" containerID="cri-o://b166cd30bea995edb16f5baec79c54bf8a533ba71167bc33c0e98da419892291" gracePeriod=30
Jul 18 17:08:17.536567 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:17.536536    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:08:17.748780 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:17.748754    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:08:17.854903 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:17.854867    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-maas-api-tls\") pod \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\" (UID: \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\") "
Jul 18 17:08:17.855064 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:17.854937    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-openshift-service-ca\") pod \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\" (UID: \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\") "
Jul 18 17:08:17.855064 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:17.854956    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvlgp\" (UniqueName: \"kubernetes.io/projected/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-kube-api-access-bvlgp\") pod \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\" (UID: \"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e\") "
Jul 18 17:08:17.855308 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:17.855286    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad8dc5a7-f33a-4c45-b5c3-389241d82d5e" (UID: "ad8dc5a7-f33a-4c45-b5c3-389241d82d5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:08:17.857283 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:17.857259    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad8dc5a7-f33a-4c45-b5c3-389241d82d5e" (UID: "ad8dc5a7-f33a-4c45-b5c3-389241d82d5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:08:17.857385 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:17.857320    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-kube-api-access-bvlgp" (OuterVolumeSpecName: "kube-api-access-bvlgp") pod "ad8dc5a7-f33a-4c45-b5c3-389241d82d5e" (UID: "ad8dc5a7-f33a-4c45-b5c3-389241d82d5e"). InnerVolumeSpecName "kube-api-access-bvlgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:08:17.955856 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:17.955818    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-maas-api-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:08:17.956014 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:17.955873    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-openshift-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:08:17.956014 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:17.955884    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvlgp\" (UniqueName: \"kubernetes.io/projected/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e-kube-api-access-bvlgp\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:08:18.557905 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:18.557870    2554 generic.go:356] "Generic (PLEG): container finished" podID="ad8dc5a7-f33a-4c45-b5c3-389241d82d5e" containerID="b166cd30bea995edb16f5baec79c54bf8a533ba71167bc33c0e98da419892291" exitCode=0
Jul 18 17:08:18.558300 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:18.557919    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr" event={"ID":"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e","Type":"ContainerDied","Data":"b166cd30bea995edb16f5baec79c54bf8a533ba71167bc33c0e98da419892291"}
Jul 18 17:08:18.558300 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:18.557955    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr" event={"ID":"ad8dc5a7-f33a-4c45-b5c3-389241d82d5e","Type":"ContainerDied","Data":"1161d305ee77c739b973a85ff1f7876a68d165afcf4f6c335f9281034cbafcbc"}
Jul 18 17:08:18.558300 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:18.557977    2554 scope.go:117] "RemoveContainer" containerID="b166cd30bea995edb16f5baec79c54bf8a533ba71167bc33c0e98da419892291"
Jul 18 17:08:18.558300 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:18.557920    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"
Jul 18 17:08:18.573957 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:18.573921    2554 scope.go:117] "RemoveContainer" containerID="b166cd30bea995edb16f5baec79c54bf8a533ba71167bc33c0e98da419892291"
Jul 18 17:08:18.574237 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:08:18.574211    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b166cd30bea995edb16f5baec79c54bf8a533ba71167bc33c0e98da419892291\": container with ID starting with b166cd30bea995edb16f5baec79c54bf8a533ba71167bc33c0e98da419892291 not found: ID does not exist" containerID="b166cd30bea995edb16f5baec79c54bf8a533ba71167bc33c0e98da419892291"
Jul 18 17:08:18.574354 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:18.574242    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b166cd30bea995edb16f5baec79c54bf8a533ba71167bc33c0e98da419892291"} err="failed to get container status \"b166cd30bea995edb16f5baec79c54bf8a533ba71167bc33c0e98da419892291\": rpc error: code = NotFound desc = could not find container \"b166cd30bea995edb16f5baec79c54bf8a533ba71167bc33c0e98da419892291\": container with ID starting with b166cd30bea995edb16f5baec79c54bf8a533ba71167bc33c0e98da419892291 not found: ID does not exist"
Jul 18 17:08:18.586371 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:18.586338    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"]
Jul 18 17:08:18.588084 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:18.588065    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2278917-859785f4bf-hw4zr"]
Jul 18 17:08:19.765597 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:19.765564    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad8dc5a7-f33a-4c45-b5c3-389241d82d5e" path="/var/lib/kubelet/pods/ad8dc5a7-f33a-4c45-b5c3-389241d82d5e/volumes"
Jul 18 17:08:22.209665 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:22.209566    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:08:22.210141 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:22.209975    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2278917-7grnx"]
Jul 18 17:08:22.214533 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:22.214509    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2278917-7grnx"]
Jul 18 17:08:22.373597 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:22.373564    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:08:22.607476 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:22.607449    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:08:22.843093 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:22.843054    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:08:22.943295 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:22.943208    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:08:23.054441 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.054407    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:08:23.398259 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.398229    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"]
Jul 18 17:08:23.398724 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.398688    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8dc5a7-f33a-4c45-b5c3-389241d82d5e" containerName="maas-api"
Jul 18 17:08:23.398724 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.398706    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8dc5a7-f33a-4c45-b5c3-389241d82d5e" containerName="maas-api"
Jul 18 17:08:23.398839 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.398741    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbcf1bc1-0848-4782-8d00-41fd8d483ad8" containerName="revoke-keys"
Jul 18 17:08:23.398839 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.398751    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbcf1bc1-0848-4782-8d00-41fd8d483ad8" containerName="revoke-keys"
Jul 18 17:08:23.398839 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.398827    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8dc5a7-f33a-4c45-b5c3-389241d82d5e" containerName="maas-api"
Jul 18 17:08:23.399024 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.398864    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbcf1bc1-0848-4782-8d00-41fd8d483ad8" containerName="revoke-keys"
Jul 18 17:08:23.401161 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.401140    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.404039 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.404021    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8e96fb11-gw-openshift-default-dockercfg-wn6g4\""
Jul 18 17:08:23.404269 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.404256    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 18 17:08:23.424138 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.424118    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"]
Jul 18 17:08:23.448592 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.448570    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:08:23.507326 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.507293    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-podinfo\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.507477 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.507341    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istiod-ca-cert\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.507477 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.507395    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5qv9\" (UniqueName: \"kubernetes.io/projected/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-kube-api-access-x5qv9\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.507549 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.507497    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-data\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.507549 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.507534    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-token\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.507617 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.507576    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-credential-socket\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.507617 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.507600    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-workload-socket\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.507680 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.507632    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-envoy\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.507680 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.507661    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-workload-certs\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.608087 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.608048    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-podinfo\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.608272 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.608099    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istiod-ca-cert\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.608272 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.608134    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5qv9\" (UniqueName: \"kubernetes.io/projected/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-kube-api-access-x5qv9\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.608272 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.608181    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-data\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.608272 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.608202    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-token\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.608493 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.608359    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-credential-socket\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.608493 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.608406    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-workload-socket\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.608493 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.608434    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-envoy\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.608493 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.608479    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-workload-certs\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.608706 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.608659    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-data\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.608797 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.608768    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-workload-socket\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.608955 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.608920    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istiod-ca-cert\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.608955 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.608943    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-credential-socket\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.609248 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.609042    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-workload-certs\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.610696 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.610675    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-podinfo\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.610781 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.610675    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-envoy\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.616546 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.616523    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-token\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.616699 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.616682    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5qv9\" (UniqueName: \"kubernetes.io/projected/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-kube-api-access-x5qv9\") pod \"e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") " pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.711216 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.711122    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:23.773889 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.772976    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbcf1bc1-0848-4782-8d00-41fd8d483ad8" path="/var/lib/kubelet/pods/fbcf1bc1-0848-4782-8d00-41fd8d483ad8/volumes"
Jul 18 17:08:23.776265 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.776236    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:08:23.850767 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:23.850743    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"]
Jul 18 17:08:23.853000 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:08:23.852975    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4ccc3e_8007_4d99_9b78_e06110a1f1d0.slice/crio-4ad438d59aff5003cbda1db8718de35987f58f98b70bea7b881de18cbbe3b8e0 WatchSource:0}: Error finding container 4ad438d59aff5003cbda1db8718de35987f58f98b70bea7b881de18cbbe3b8e0: Status 404 returned error can't find the container with id 4ad438d59aff5003cbda1db8718de35987f58f98b70bea7b881de18cbbe3b8e0
Jul 18 17:08:24.586726 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:24.586679    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6" event={"ID":"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0","Type":"ContainerStarted","Data":"4ad438d59aff5003cbda1db8718de35987f58f98b70bea7b881de18cbbe3b8e0"}
Jul 18 17:08:26.361668 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:26.361625    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:08:26.361960 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:26.361708    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:08:26.361960 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:26.361741    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:08:26.598078 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:26.598040    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6" event={"ID":"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0","Type":"ContainerStarted","Data":"cdeadac0a6c87927370cd983820978cee5c91b0200933f581ecbe11c973afea5"}
Jul 18 17:08:26.618957 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:26.618838    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6" podStartSLOduration=1.112610778 podStartE2EDuration="3.61881541s" totalImagesPullingTime="2.506204632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:08:26.616424847 +0000 UTC m=+1925.314255532" watchObservedRunningTime="2026-07-18 17:08:26.61881541 +0000 UTC m=+1925.316646100"
Jul 18 17:08:26.711628 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:26.711598    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:27.717096 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:27.717067    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:28.063361 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.063276    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:08:28.144425 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.144350    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"]
Jul 18 17:08:28.149109 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.149083    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:08:28.150578 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.150551    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"]
Jul 18 17:08:28.152008 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.151982    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 18 17:08:28.152123 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.152052    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 18 17:08:28.152185 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.152139    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2pqsv\""
Jul 18 17:08:28.152185 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.151982    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8e96fb11-serving-cert\""
Jul 18 17:08:28.169283 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.169260    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:08:28.248593 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.248554    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5wwb\" (UniqueName: \"kubernetes.io/projected/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-kube-api-access-w5wwb\") pod \"maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9\" (UID: \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:08:28.248593 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.248592    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-openshift-service-ca\") pod \"maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9\" (UID: \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:08:28.248874 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.248627    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-maas-api-tls\") pod \"maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9\" (UID: \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:08:28.350004 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.349971    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5wwb\" (UniqueName: \"kubernetes.io/projected/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-kube-api-access-w5wwb\") pod \"maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9\" (UID: \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:08:28.350160 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.350014    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-openshift-service-ca\") pod \"maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9\" (UID: \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:08:28.350160 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.350043    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-maas-api-tls\") pod \"maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9\" (UID: \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:08:28.350231 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:08:28.350169    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-serving-cert: secret "maas-api-e2e-migrate-8e96fb11-serving-cert" not found
Jul 18 17:08:28.350231 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:08:28.350215    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-maas-api-tls podName:f3b6a75a-b1db-4c89-aae5-9d88b88d176d nodeName:}" failed. No retries permitted until 2026-07-18 17:08:28.850200149 +0000 UTC m=+1927.548030812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-maas-api-tls") pod "maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9" (UID: "f3b6a75a-b1db-4c89-aae5-9d88b88d176d") : secret "maas-api-e2e-migrate-8e96fb11-serving-cert" not found
Jul 18 17:08:28.350628 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.350610    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-openshift-service-ca\") pod \"maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9\" (UID: \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:08:28.360463 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.360439    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5wwb\" (UniqueName: \"kubernetes.io/projected/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-kube-api-access-w5wwb\") pod \"maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9\" (UID: \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:08:28.606461 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.606376    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:28.607465 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.607445    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:28.613576 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.613549    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:08:28.749651 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.749615    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:08:28.852390 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.852346    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:08:28.854606 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.854585    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-maas-api-tls\") pod \"maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9\" (UID: \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:08:28.857275 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:28.857222    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-maas-api-tls\") pod \"maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9\" (UID: \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:08:29.060499 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:29.060466    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:08:29.211927 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:29.211889    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"]
Jul 18 17:08:29.214003 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:08:29.213979    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b6a75a_b1db_4c89_aae5_9d88b88d176d.slice/crio-ad327c1b3bf5182f2d72e04d1e56a4db821d3da42e18cb334fec32d990a70452 WatchSource:0}: Error finding container ad327c1b3bf5182f2d72e04d1e56a4db821d3da42e18cb334fec32d990a70452: Status 404 returned error can't find the container with id ad327c1b3bf5182f2d72e04d1e56a4db821d3da42e18cb334fec32d990a70452
Jul 18 17:08:29.611447 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:29.611420    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9" event={"ID":"f3b6a75a-b1db-4c89-aae5-9d88b88d176d","Type":"ContainerStarted","Data":"ad327c1b3bf5182f2d72e04d1e56a4db821d3da42e18cb334fec32d990a70452"}
Jul 18 17:08:30.618032 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:30.617997    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9" event={"ID":"f3b6a75a-b1db-4c89-aae5-9d88b88d176d","Type":"ContainerStarted","Data":"8aed20a82a7d18d6e69abc755f859e9df315ece59dd17aa0ba4199bf18e5e24c"}
Jul 18 17:08:30.618450 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:30.618247    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:08:30.634605 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:30.634559    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9" podStartSLOduration=2.244892559 podStartE2EDuration="2.634543573s" totalImagesPullingTime="389.651014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:08:30.633573474 +0000 UTC m=+1929.331404160" watchObservedRunningTime="2026-07-18 17:08:30.634543573 +0000 UTC m=+1929.332374258"
Jul 18 17:08:36.628410 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:36.628379    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:08:40.663459 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:40.663428    2554 generic.go:356] "Generic (PLEG): container finished" podID="1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" containerID="cdeadac0a6c87927370cd983820978cee5c91b0200933f581ecbe11c973afea5" exitCode=0
Jul 18 17:08:40.663891 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:40.663506    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6" event={"ID":"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0","Type":"ContainerDied","Data":"cdeadac0a6c87927370cd983820978cee5c91b0200933f581ecbe11c973afea5"}
Jul 18 17:08:40.663938 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:40.663890    2554 scope.go:117] "RemoveContainer" containerID="cdeadac0a6c87927370cd983820978cee5c91b0200933f581ecbe11c973afea5"
Jul 18 17:08:40.665919 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:40.665875    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:08:40.666036 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:40.665951    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:08:40.666036 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:40.665987    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:08:41.672926 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:41.672883    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6" event={"ID":"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0","Type":"ContainerStarted","Data":"2dc2effb1fa88338e26148df9c2e4dc91f41504cd4f68a8e6168485e86c2e182"}
Jul 18 17:08:41.711264 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:41.711231    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:41.712245 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:41.712219    2554 patch_prober.go:28] interesting pod/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 18 17:08:41.712349 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:41.712275    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6" podUID="1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 18 17:08:42.712064 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:42.712026    2554 patch_prober.go:28] interesting pod/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 18 17:08:42.712455 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:42.712084    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6" podUID="1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 18 17:08:43.607280 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:43.607244    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:43.711637 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:43.711598    2554 patch_prober.go:28] interesting pod/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 18 17:08:43.711796 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:43.711672    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6" podUID="1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 18 17:08:44.712211 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:44.712174    2554 patch_prober.go:28] interesting pod/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 18 17:08:44.712674 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:44.712230    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6" podUID="1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 18 17:08:45.715346 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:45.715316    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:45.716227 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:45.716209    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:08:53.040314 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:53.040278    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg"]
Jul 18 17:08:53.043633 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:53.043615    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg"
Jul 18 17:08:53.046070 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:53.046048    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7rq8\""
Jul 18 17:08:53.050389 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:53.050353    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg"]
Jul 18 17:08:53.164548 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:53.164507    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c45b049f-8501-4eda-b862-39da70cf194b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg\" (UID: \"c45b049f-8501-4eda-b862-39da70cf194b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg"
Jul 18 17:08:53.265944 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:53.265905    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c45b049f-8501-4eda-b862-39da70cf194b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg\" (UID: \"c45b049f-8501-4eda-b862-39da70cf194b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg"
Jul 18 17:08:53.266696 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:53.266672    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c45b049f-8501-4eda-b862-39da70cf194b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg\" (UID: \"c45b049f-8501-4eda-b862-39da70cf194b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg"
Jul 18 17:08:53.354532 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:53.354493    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg"
Jul 18 17:08:53.482382 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:53.482347    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg"]
Jul 18 17:08:53.484749 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:08:53.484721    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45b049f_8501_4eda_b862_39da70cf194b.slice/crio-95641d5680db1b9b70a696d9573136a198208c40dd891cb4d042ebc6ae990fa4 WatchSource:0}: Error finding container 95641d5680db1b9b70a696d9573136a198208c40dd891cb4d042ebc6ae990fa4: Status 404 returned error can't find the container with id 95641d5680db1b9b70a696d9573136a198208c40dd891cb4d042ebc6ae990fa4
Jul 18 17:08:53.737909 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:53.737875    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg" event={"ID":"c45b049f-8501-4eda-b862-39da70cf194b","Type":"ContainerStarted","Data":"95641d5680db1b9b70a696d9573136a198208c40dd891cb4d042ebc6ae990fa4"}
Jul 18 17:08:54.744075 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:54.744043    2554 generic.go:356] "Generic (PLEG): container finished" podID="c45b049f-8501-4eda-b862-39da70cf194b" containerID="04d23bed89052ee22515459e59341e9e71a6e77b922043c5bf7b022441d94996" exitCode=0
Jul 18 17:08:54.744478 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:54.744093    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg" event={"ID":"c45b049f-8501-4eda-b862-39da70cf194b","Type":"ContainerDied","Data":"04d23bed89052ee22515459e59341e9e71a6e77b922043c5bf7b022441d94996"}
Jul 18 17:08:55.884003 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:55.883978    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg"
Jul 18 17:08:55.991005 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:55.990966    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c45b049f-8501-4eda-b862-39da70cf194b-maas-api-service-ca\") pod \"c45b049f-8501-4eda-b862-39da70cf194b\" (UID: \"c45b049f-8501-4eda-b862-39da70cf194b\") "
Jul 18 17:08:55.991317 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:55.991294    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45b049f-8501-4eda-b862-39da70cf194b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c45b049f-8501-4eda-b862-39da70cf194b" (UID: "c45b049f-8501-4eda-b862-39da70cf194b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:08:56.091802 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:56.091774    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c45b049f-8501-4eda-b862-39da70cf194b-maas-api-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:08:56.753979 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:56.753938    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg" event={"ID":"c45b049f-8501-4eda-b862-39da70cf194b","Type":"ContainerDied","Data":"95641d5680db1b9b70a696d9573136a198208c40dd891cb4d042ebc6ae990fa4"}
Jul 18 17:08:56.754152 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:56.753990    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95641d5680db1b9b70a696d9573136a198208c40dd891cb4d042ebc6ae990fa4"
Jul 18 17:08:56.754152 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:08:56.753957    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg"
Jul 18 17:09:03.422477 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.422440    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"]
Jul 18 17:09:03.423074 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.422735    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9" podUID="f3b6a75a-b1db-4c89-aae5-9d88b88d176d" containerName="maas-api" containerID="cri-o://8aed20a82a7d18d6e69abc755f859e9df315ece59dd17aa0ba4199bf18e5e24c" gracePeriod=30
Jul 18 17:09:03.446048 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.446018    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:03.673095 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.673031    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:09:03.763864 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.763821    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-maas-api-tls\") pod \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\" (UID: \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\") "
Jul 18 17:09:03.764057 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.763936    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5wwb\" (UniqueName: \"kubernetes.io/projected/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-kube-api-access-w5wwb\") pod \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\" (UID: \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\") "
Jul 18 17:09:03.764057 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.763973    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-openshift-service-ca\") pod \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\" (UID: \"f3b6a75a-b1db-4c89-aae5-9d88b88d176d\") "
Jul 18 17:09:03.764386 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.764360    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3b6a75a-b1db-4c89-aae5-9d88b88d176d" (UID: "f3b6a75a-b1db-4c89-aae5-9d88b88d176d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:09:03.766355 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.766326    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-kube-api-access-w5wwb" (OuterVolumeSpecName: "kube-api-access-w5wwb") pod "f3b6a75a-b1db-4c89-aae5-9d88b88d176d" (UID: "f3b6a75a-b1db-4c89-aae5-9d88b88d176d"). InnerVolumeSpecName "kube-api-access-w5wwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:09:03.766502 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.766389    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3b6a75a-b1db-4c89-aae5-9d88b88d176d" (UID: "f3b6a75a-b1db-4c89-aae5-9d88b88d176d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:09:03.785659 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.785629    2554 generic.go:356] "Generic (PLEG): container finished" podID="f3b6a75a-b1db-4c89-aae5-9d88b88d176d" containerID="8aed20a82a7d18d6e69abc755f859e9df315ece59dd17aa0ba4199bf18e5e24c" exitCode=0
Jul 18 17:09:03.785778 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.785678    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"
Jul 18 17:09:03.785778 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.785703    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9" event={"ID":"f3b6a75a-b1db-4c89-aae5-9d88b88d176d","Type":"ContainerDied","Data":"8aed20a82a7d18d6e69abc755f859e9df315ece59dd17aa0ba4199bf18e5e24c"}
Jul 18 17:09:03.785778 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.785753    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9" event={"ID":"f3b6a75a-b1db-4c89-aae5-9d88b88d176d","Type":"ContainerDied","Data":"ad327c1b3bf5182f2d72e04d1e56a4db821d3da42e18cb334fec32d990a70452"}
Jul 18 17:09:03.785778 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.785774    2554 scope.go:117] "RemoveContainer" containerID="8aed20a82a7d18d6e69abc755f859e9df315ece59dd17aa0ba4199bf18e5e24c"
Jul 18 17:09:03.794369 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.794349    2554 scope.go:117] "RemoveContainer" containerID="8aed20a82a7d18d6e69abc755f859e9df315ece59dd17aa0ba4199bf18e5e24c"
Jul 18 17:09:03.794632 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:09:03.794613    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aed20a82a7d18d6e69abc755f859e9df315ece59dd17aa0ba4199bf18e5e24c\": container with ID starting with 8aed20a82a7d18d6e69abc755f859e9df315ece59dd17aa0ba4199bf18e5e24c not found: ID does not exist" containerID="8aed20a82a7d18d6e69abc755f859e9df315ece59dd17aa0ba4199bf18e5e24c"
Jul 18 17:09:03.794684 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.794649    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aed20a82a7d18d6e69abc755f859e9df315ece59dd17aa0ba4199bf18e5e24c"} err="failed to get container status \"8aed20a82a7d18d6e69abc755f859e9df315ece59dd17aa0ba4199bf18e5e24c\": rpc error: code = NotFound desc = could not find container \"8aed20a82a7d18d6e69abc755f859e9df315ece59dd17aa0ba4199bf18e5e24c\": container with ID starting with 8aed20a82a7d18d6e69abc755f859e9df315ece59dd17aa0ba4199bf18e5e24c not found: ID does not exist"
Jul 18 17:09:03.801899 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.801868    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"]
Jul 18 17:09:03.807774 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.807745    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8e96fb11-9b867f6c-nd2q9"]
Jul 18 17:09:03.865388 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.865348    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5wwb\" (UniqueName: \"kubernetes.io/projected/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-kube-api-access-w5wwb\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:03.865388 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.865384    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-openshift-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:03.865388 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:03.865394    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b6a75a-b1db-4c89-aae5-9d88b88d176d-maas-api-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:05.766583 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:05.766551    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3b6a75a-b1db-4c89-aae5-9d88b88d176d" path="/var/lib/kubelet/pods/f3b6a75a-b1db-4c89-aae5-9d88b88d176d/volumes"
Jul 18 17:09:08.120789 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:08.120752    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:08.127338 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:08.127310    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg"]
Jul 18 17:09:08.129330 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:08.129309    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8e96fb11-pzppg"]
Jul 18 17:09:08.263776 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:08.263746    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"]
Jul 18 17:09:08.264047 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:08.264004    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6" podUID="1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" containerName="istio-proxy" containerID="cri-o://2dc2effb1fa88338e26148df9c2e4dc91f41504cd4f68a8e6168485e86c2e182" gracePeriod=30
Jul 18 17:09:08.295708 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:08.295682    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:08.515956 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:08.515877    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:08.750286 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:08.750251    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:09.044895 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:09.044864    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:09.283492 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:09.283452    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:09.766615 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:09.766579    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c45b049f-8501-4eda-b862-39da70cf194b" path="/var/lib/kubelet/pods/c45b049f-8501-4eda-b862-39da70cf194b/volumes"
Jul 18 17:09:11.038688 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.038657    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:11.140048 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.140017    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:11.265863 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.265813    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"]
Jul 18 17:09:11.266201 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.266179    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b6a75a-b1db-4c89-aae5-9d88b88d176d" containerName="maas-api"
Jul 18 17:09:11.266201 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.266201    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b6a75a-b1db-4c89-aae5-9d88b88d176d" containerName="maas-api"
Jul 18 17:09:11.266360 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.266215    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c45b049f-8501-4eda-b862-39da70cf194b" containerName="revoke-keys"
Jul 18 17:09:11.266360 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.266221    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c45b049f-8501-4eda-b862-39da70cf194b" containerName="revoke-keys"
Jul 18 17:09:11.266360 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.266290    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c45b049f-8501-4eda-b862-39da70cf194b" containerName="revoke-keys"
Jul 18 17:09:11.266360 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.266297    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b6a75a-b1db-4c89-aae5-9d88b88d176d" containerName="maas-api"
Jul 18 17:09:11.270453 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.270436    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"
Jul 18 17:09:11.273045 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.273028    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 18 17:09:11.273127 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.273029    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 18 17:09:11.273910 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.273892    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b26f5ca7-serving-cert\""
Jul 18 17:09:11.273963 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.273896    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2pqsv\""
Jul 18 17:09:11.278016 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.277969    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"]
Jul 18 17:09:11.298919 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.298830    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:11.432338 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.432302    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa122bd-abbf-4700-97bc-bcc417369f45-maas-api-tls\") pod \"maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp\" (UID: \"1aa122bd-abbf-4700-97bc-bcc417369f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"
Jul 18 17:09:11.432499 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.432345    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxq2r\" (UniqueName: \"kubernetes.io/projected/1aa122bd-abbf-4700-97bc-bcc417369f45-kube-api-access-sxq2r\") pod \"maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp\" (UID: \"1aa122bd-abbf-4700-97bc-bcc417369f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"
Jul 18 17:09:11.432499 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.432371    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa122bd-abbf-4700-97bc-bcc417369f45-openshift-service-ca\") pod \"maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp\" (UID: \"1aa122bd-abbf-4700-97bc-bcc417369f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"
Jul 18 17:09:11.501035 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.501005    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:11.533088 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.533052    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa122bd-abbf-4700-97bc-bcc417369f45-openshift-service-ca\") pod \"maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp\" (UID: \"1aa122bd-abbf-4700-97bc-bcc417369f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"
Jul 18 17:09:11.533259 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.533183    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa122bd-abbf-4700-97bc-bcc417369f45-maas-api-tls\") pod \"maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp\" (UID: \"1aa122bd-abbf-4700-97bc-bcc417369f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"
Jul 18 17:09:11.533259 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.533224    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxq2r\" (UniqueName: \"kubernetes.io/projected/1aa122bd-abbf-4700-97bc-bcc417369f45-kube-api-access-sxq2r\") pod \"maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp\" (UID: \"1aa122bd-abbf-4700-97bc-bcc417369f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"
Jul 18 17:09:11.533734 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.533711    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa122bd-abbf-4700-97bc-bcc417369f45-openshift-service-ca\") pod \"maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp\" (UID: \"1aa122bd-abbf-4700-97bc-bcc417369f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"
Jul 18 17:09:11.535764 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.535743    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa122bd-abbf-4700-97bc-bcc417369f45-maas-api-tls\") pod \"maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp\" (UID: \"1aa122bd-abbf-4700-97bc-bcc417369f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"
Jul 18 17:09:11.541838 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.541813    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxq2r\" (UniqueName: \"kubernetes.io/projected/1aa122bd-abbf-4700-97bc-bcc417369f45-kube-api-access-sxq2r\") pod \"maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp\" (UID: \"1aa122bd-abbf-4700-97bc-bcc417369f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"
Jul 18 17:09:11.582296 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.582266    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"
Jul 18 17:09:11.719572 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.719547    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"]
Jul 18 17:09:11.722382 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:09:11.722353    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa122bd_abbf_4700_97bc_bcc417369f45.slice/crio-74b9f208e012ee6eb152def851a6659bb0bd0d21e09dd5144fd04d1216c72930 WatchSource:0}: Error finding container 74b9f208e012ee6eb152def851a6659bb0bd0d21e09dd5144fd04d1216c72930: Status 404 returned error can't find the container with id 74b9f208e012ee6eb152def851a6659bb0bd0d21e09dd5144fd04d1216c72930
Jul 18 17:09:11.826642 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.826604    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp" event={"ID":"1aa122bd-abbf-4700-97bc-bcc417369f45","Type":"ContainerStarted","Data":"74b9f208e012ee6eb152def851a6659bb0bd0d21e09dd5144fd04d1216c72930"}
Jul 18 17:09:11.987593 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:11.987501    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:12.077305 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:12.077267    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:12.833237 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:12.833152    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp" event={"ID":"1aa122bd-abbf-4700-97bc-bcc417369f45","Type":"ContainerStarted","Data":"7e6ec3e9885fafa93add9a75ab99c71bd5e0064d5c60687a19f90ad50afdcda2"}
Jul 18 17:09:12.833237 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:12.833220    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"
Jul 18 17:09:12.878868 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:12.878802    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp" podStartSLOduration=1.306500125 podStartE2EDuration="1.878786409s" totalImagesPullingTime="572.286284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:09:12.877623916 +0000 UTC m=+1971.575454600" watchObservedRunningTime="2026-07-18 17:09:12.878786409 +0000 UTC m=+1971.576617092"
Jul 18 17:09:13.518107 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.518085    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:09:13.653081 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.653054    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-podinfo\") pod \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") "
Jul 18 17:09:13.653281 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.653094    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5qv9\" (UniqueName: \"kubernetes.io/projected/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-kube-api-access-x5qv9\") pod \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") "
Jul 18 17:09:13.653281 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.653117    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-workload-certs\") pod \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") "
Jul 18 17:09:13.653281 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.653139    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-workload-socket\") pod \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") "
Jul 18 17:09:13.653281 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.653180    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istiod-ca-cert\") pod \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") "
Jul 18 17:09:13.653281 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.653197    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-credential-socket\") pod \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") "
Jul 18 17:09:13.653281 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.653215    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-data\") pod \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") "
Jul 18 17:09:13.653281 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.653247    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-token\") pod \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") "
Jul 18 17:09:13.653661 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.653284    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-envoy\") pod \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\" (UID: \"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0\") "
Jul 18 17:09:13.653661 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.653446    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" (UID: "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:09:13.653661 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.653511    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" (UID: "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:09:13.653661 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.653539    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" (UID: "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:09:13.653661 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.653612    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" (UID: "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:09:13.654759 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.654724    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" (UID: "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:09:13.655580 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.655552    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" (UID: "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 17:09:13.655678 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.655575    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" (UID: "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:09:13.655678 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.655607    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-kube-api-access-x5qv9" (OuterVolumeSpecName: "kube-api-access-x5qv9") pod "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" (UID: "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0"). InnerVolumeSpecName "kube-api-access-x5qv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:09:13.655678 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.655657    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" (UID: "1f4ccc3e-8007-4d99-9b78-e06110a1f1d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:09:13.754308 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.754255    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5qv9\" (UniqueName: \"kubernetes.io/projected/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-kube-api-access-x5qv9\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:13.754308 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.754301    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-workload-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:13.754308 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.754313    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-workload-socket\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:13.754308 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.754320    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istiod-ca-cert\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:13.754576 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.754329    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-credential-socket\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:13.754576 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.754342    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-data\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:13.754576 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.754350    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-token\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:13.754576 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.754358    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-envoy\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:13.754576 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.754365    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0-istio-podinfo\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:13.839370 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.839339    2554 generic.go:356] "Generic (PLEG): container finished" podID="1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" containerID="2dc2effb1fa88338e26148df9c2e4dc91f41504cd4f68a8e6168485e86c2e182" exitCode=0
Jul 18 17:09:13.839531 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.839388    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"
Jul 18 17:09:13.839531 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.839428    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6" event={"ID":"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0","Type":"ContainerDied","Data":"2dc2effb1fa88338e26148df9c2e4dc91f41504cd4f68a8e6168485e86c2e182"}
Jul 18 17:09:13.839531 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.839481    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6" event={"ID":"1f4ccc3e-8007-4d99-9b78-e06110a1f1d0","Type":"ContainerDied","Data":"4ad438d59aff5003cbda1db8718de35987f58f98b70bea7b881de18cbbe3b8e0"}
Jul 18 17:09:13.839531 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.839504    2554 scope.go:117] "RemoveContainer" containerID="2dc2effb1fa88338e26148df9c2e4dc91f41504cd4f68a8e6168485e86c2e182"
Jul 18 17:09:13.847989 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.847973    2554 scope.go:117] "RemoveContainer" containerID="cdeadac0a6c87927370cd983820978cee5c91b0200933f581ecbe11c973afea5"
Jul 18 17:09:13.856177 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.856138    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"]
Jul 18 17:09:13.856230 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.856202    2554 scope.go:117] "RemoveContainer" containerID="2dc2effb1fa88338e26148df9c2e4dc91f41504cd4f68a8e6168485e86c2e182"
Jul 18 17:09:13.856465 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:09:13.856446    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc2effb1fa88338e26148df9c2e4dc91f41504cd4f68a8e6168485e86c2e182\": container with ID starting with 2dc2effb1fa88338e26148df9c2e4dc91f41504cd4f68a8e6168485e86c2e182 not found: ID does not exist" containerID="2dc2effb1fa88338e26148df9c2e4dc91f41504cd4f68a8e6168485e86c2e182"
Jul 18 17:09:13.856516 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.856474    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc2effb1fa88338e26148df9c2e4dc91f41504cd4f68a8e6168485e86c2e182"} err="failed to get container status \"2dc2effb1fa88338e26148df9c2e4dc91f41504cd4f68a8e6168485e86c2e182\": rpc error: code = NotFound desc = could not find container \"2dc2effb1fa88338e26148df9c2e4dc91f41504cd4f68a8e6168485e86c2e182\": container with ID starting with 2dc2effb1fa88338e26148df9c2e4dc91f41504cd4f68a8e6168485e86c2e182 not found: ID does not exist"
Jul 18 17:09:13.856516 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.856488    2554 scope.go:117] "RemoveContainer" containerID="cdeadac0a6c87927370cd983820978cee5c91b0200933f581ecbe11c973afea5"
Jul 18 17:09:13.856734 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:09:13.856720    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdeadac0a6c87927370cd983820978cee5c91b0200933f581ecbe11c973afea5\": container with ID starting with cdeadac0a6c87927370cd983820978cee5c91b0200933f581ecbe11c973afea5 not found: ID does not exist" containerID="cdeadac0a6c87927370cd983820978cee5c91b0200933f581ecbe11c973afea5"
Jul 18 17:09:13.856777 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.856738    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdeadac0a6c87927370cd983820978cee5c91b0200933f581ecbe11c973afea5"} err="failed to get container status \"cdeadac0a6c87927370cd983820978cee5c91b0200933f581ecbe11c973afea5\": rpc error: code = NotFound desc = could not find container \"cdeadac0a6c87927370cd983820978cee5c91b0200933f581ecbe11c973afea5\": container with ID starting with cdeadac0a6c87927370cd983820978cee5c91b0200933f581ecbe11c973afea5 not found: ID does not exist"
Jul 18 17:09:13.859302 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:13.859281    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8e96fb11-gw-openshift-default-6754b99f99-hzkz6"]
Jul 18 17:09:15.766237 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:15.766206    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" path="/var/lib/kubelet/pods/1f4ccc3e-8007-4d99-9b78-e06110a1f1d0/volumes"
Jul 18 17:09:18.845443 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:18.845412    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"
Jul 18 17:09:22.522900 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:22.522869    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:36.799896 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:36.799857    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f"]
Jul 18 17:09:36.800387 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:36.800257    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" containerName="istio-proxy"
Jul 18 17:09:36.800387 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:36.800275    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" containerName="istio-proxy"
Jul 18 17:09:36.800387 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:36.800294    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" containerName="istio-proxy"
Jul 18 17:09:36.800387 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:36.800303    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" containerName="istio-proxy"
Jul 18 17:09:36.800387 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:36.800374    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" containerName="istio-proxy"
Jul 18 17:09:36.800387 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:36.800385    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f4ccc3e-8007-4d99-9b78-e06110a1f1d0" containerName="istio-proxy"
Jul 18 17:09:36.803395 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:36.803380    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f"
Jul 18 17:09:36.806015 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:36.805995    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7rq8\""
Jul 18 17:09:36.810679 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:36.810321    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f"]
Jul 18 17:09:36.855440 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:36.855411    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3525897-7911-4738-bc82-246b7db1befd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f\" (UID: \"d3525897-7911-4738-bc82-246b7db1befd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f"
Jul 18 17:09:36.956220 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:36.956191    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3525897-7911-4738-bc82-246b7db1befd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f\" (UID: \"d3525897-7911-4738-bc82-246b7db1befd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f"
Jul 18 17:09:36.956738 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:36.956721    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3525897-7911-4738-bc82-246b7db1befd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f\" (UID: \"d3525897-7911-4738-bc82-246b7db1befd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f"
Jul 18 17:09:37.115181 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:37.115146    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f"
Jul 18 17:09:37.243774 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:37.243748    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f"]
Jul 18 17:09:37.245968 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:09:37.245942    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3525897_7911_4738_bc82_246b7db1befd.slice/crio-151fb127615a519104c7f979fd52c30781756e241d841a1e6ec28e3f5e61fdae WatchSource:0}: Error finding container 151fb127615a519104c7f979fd52c30781756e241d841a1e6ec28e3f5e61fdae: Status 404 returned error can't find the container with id 151fb127615a519104c7f979fd52c30781756e241d841a1e6ec28e3f5e61fdae
Jul 18 17:09:37.953535 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:37.953500    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f" event={"ID":"d3525897-7911-4738-bc82-246b7db1befd","Type":"ContainerStarted","Data":"206b2639aa2e52cbc8e9c20bdc9fd4bc67a76119ab44fdcbd875fe2aaec3f03d"}
Jul 18 17:09:37.953947 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:37.953555    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f" event={"ID":"d3525897-7911-4738-bc82-246b7db1befd","Type":"ContainerStarted","Data":"151fb127615a519104c7f979fd52c30781756e241d841a1e6ec28e3f5e61fdae"}
Jul 18 17:09:37.969812 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:37.969766    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f" podStartSLOduration=1.969752261 podStartE2EDuration="1.969752261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:09:37.968280313 +0000 UTC m=+1996.666110997" watchObservedRunningTime="2026-07-18 17:09:37.969752261 +0000 UTC m=+1996.667582945"
Jul 18 17:09:38.959516 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:38.959484    2554 generic.go:356] "Generic (PLEG): container finished" podID="d3525897-7911-4738-bc82-246b7db1befd" containerID="206b2639aa2e52cbc8e9c20bdc9fd4bc67a76119ab44fdcbd875fe2aaec3f03d" exitCode=0
Jul 18 17:09:38.959943 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:38.959540    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f" event={"ID":"d3525897-7911-4738-bc82-246b7db1befd","Type":"ContainerDied","Data":"206b2639aa2e52cbc8e9c20bdc9fd4bc67a76119ab44fdcbd875fe2aaec3f03d"}
Jul 18 17:09:40.090484 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:40.090458    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f"
Jul 18 17:09:40.182949 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:40.182916    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3525897-7911-4738-bc82-246b7db1befd-maas-api-service-ca\") pod \"d3525897-7911-4738-bc82-246b7db1befd\" (UID: \"d3525897-7911-4738-bc82-246b7db1befd\") "
Jul 18 17:09:40.183281 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:40.183256    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3525897-7911-4738-bc82-246b7db1befd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3525897-7911-4738-bc82-246b7db1befd" (UID: "d3525897-7911-4738-bc82-246b7db1befd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:09:40.284203 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:40.284121    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3525897-7911-4738-bc82-246b7db1befd-maas-api-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:40.969227 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:40.969203    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f"
Jul 18 17:09:40.969386 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:40.969226    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f" event={"ID":"d3525897-7911-4738-bc82-246b7db1befd","Type":"ContainerDied","Data":"151fb127615a519104c7f979fd52c30781756e241d841a1e6ec28e3f5e61fdae"}
Jul 18 17:09:40.969386 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:40.969270    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="151fb127615a519104c7f979fd52c30781756e241d841a1e6ec28e3f5e61fdae"
Jul 18 17:09:47.210328 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:47.210283    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"]
Jul 18 17:09:47.210800 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:47.210520    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp" podUID="1aa122bd-abbf-4700-97bc-bcc417369f45" containerName="maas-api" containerID="cri-o://7e6ec3e9885fafa93add9a75ab99c71bd5e0064d5c60687a19f90ad50afdcda2" gracePeriod=30
Jul 18 17:09:47.237883 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:47.237834    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:47.456294 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:47.456271    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"
Jul 18 17:09:47.548460 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:47.548371    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa122bd-abbf-4700-97bc-bcc417369f45-maas-api-tls\") pod \"1aa122bd-abbf-4700-97bc-bcc417369f45\" (UID: \"1aa122bd-abbf-4700-97bc-bcc417369f45\") "
Jul 18 17:09:47.548594 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:47.548471    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxq2r\" (UniqueName: \"kubernetes.io/projected/1aa122bd-abbf-4700-97bc-bcc417369f45-kube-api-access-sxq2r\") pod \"1aa122bd-abbf-4700-97bc-bcc417369f45\" (UID: \"1aa122bd-abbf-4700-97bc-bcc417369f45\") "
Jul 18 17:09:47.548594 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:47.548584    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa122bd-abbf-4700-97bc-bcc417369f45-openshift-service-ca\") pod \"1aa122bd-abbf-4700-97bc-bcc417369f45\" (UID: \"1aa122bd-abbf-4700-97bc-bcc417369f45\") "
Jul 18 17:09:47.548984 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:47.548961    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa122bd-abbf-4700-97bc-bcc417369f45-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1aa122bd-abbf-4700-97bc-bcc417369f45" (UID: "1aa122bd-abbf-4700-97bc-bcc417369f45"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:09:47.550724 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:47.550697    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa122bd-abbf-4700-97bc-bcc417369f45-kube-api-access-sxq2r" (OuterVolumeSpecName: "kube-api-access-sxq2r") pod "1aa122bd-abbf-4700-97bc-bcc417369f45" (UID: "1aa122bd-abbf-4700-97bc-bcc417369f45"). InnerVolumeSpecName "kube-api-access-sxq2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:09:47.550724 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:47.550702    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa122bd-abbf-4700-97bc-bcc417369f45-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1aa122bd-abbf-4700-97bc-bcc417369f45" (UID: "1aa122bd-abbf-4700-97bc-bcc417369f45"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:09:47.649943 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:47.649906    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa122bd-abbf-4700-97bc-bcc417369f45-openshift-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:47.649943 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:47.649936    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa122bd-abbf-4700-97bc-bcc417369f45-maas-api-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:47.649943 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:47.649949    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxq2r\" (UniqueName: \"kubernetes.io/projected/1aa122bd-abbf-4700-97bc-bcc417369f45-kube-api-access-sxq2r\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:09:48.000697 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:48.000662    2554 generic.go:356] "Generic (PLEG): container finished" podID="1aa122bd-abbf-4700-97bc-bcc417369f45" containerID="7e6ec3e9885fafa93add9a75ab99c71bd5e0064d5c60687a19f90ad50afdcda2" exitCode=0
Jul 18 17:09:48.000882 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:48.000709    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"
Jul 18 17:09:48.000882 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:48.000751    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp" event={"ID":"1aa122bd-abbf-4700-97bc-bcc417369f45","Type":"ContainerDied","Data":"7e6ec3e9885fafa93add9a75ab99c71bd5e0064d5c60687a19f90ad50afdcda2"}
Jul 18 17:09:48.000882 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:48.000804    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp" event={"ID":"1aa122bd-abbf-4700-97bc-bcc417369f45","Type":"ContainerDied","Data":"74b9f208e012ee6eb152def851a6659bb0bd0d21e09dd5144fd04d1216c72930"}
Jul 18 17:09:48.000882 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:48.000830    2554 scope.go:117] "RemoveContainer" containerID="7e6ec3e9885fafa93add9a75ab99c71bd5e0064d5c60687a19f90ad50afdcda2"
Jul 18 17:09:48.009248 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:48.009230    2554 scope.go:117] "RemoveContainer" containerID="7e6ec3e9885fafa93add9a75ab99c71bd5e0064d5c60687a19f90ad50afdcda2"
Jul 18 17:09:48.009504 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:09:48.009479    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6ec3e9885fafa93add9a75ab99c71bd5e0064d5c60687a19f90ad50afdcda2\": container with ID starting with 7e6ec3e9885fafa93add9a75ab99c71bd5e0064d5c60687a19f90ad50afdcda2 not found: ID does not exist" containerID="7e6ec3e9885fafa93add9a75ab99c71bd5e0064d5c60687a19f90ad50afdcda2"
Jul 18 17:09:48.009586 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:48.009515    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6ec3e9885fafa93add9a75ab99c71bd5e0064d5c60687a19f90ad50afdcda2"} err="failed to get container status \"7e6ec3e9885fafa93add9a75ab99c71bd5e0064d5c60687a19f90ad50afdcda2\": rpc error: code = NotFound desc = could not find container \"7e6ec3e9885fafa93add9a75ab99c71bd5e0064d5c60687a19f90ad50afdcda2\": container with ID starting with 7e6ec3e9885fafa93add9a75ab99c71bd5e0064d5c60687a19f90ad50afdcda2 not found: ID does not exist"
Jul 18 17:09:48.017195 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:48.017140    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"]
Jul 18 17:09:48.018763 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:48.018746    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b26f5ca7-77b47744f8-xxslp"]
Jul 18 17:09:49.765372 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:49.765344    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aa122bd-abbf-4700-97bc-bcc417369f45" path="/var/lib/kubelet/pods/1aa122bd-abbf-4700-97bc-bcc417369f45/volumes"
Jul 18 17:09:51.874186 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:51.874151    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:51.874820 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:51.874780    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f"]
Jul 18 17:09:51.876328 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:51.876309    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b26f5ca7-jft5f"]
Jul 18 17:09:53.049436 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.049405    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:53.279173 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.279132    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:53.506501 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.506463    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:53.766434 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.766352    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3525897-7911-4738-bc82-246b7db1befd" path="/var/lib/kubelet/pods/d3525897-7911-4738-bc82-246b7db1befd/volumes"
Jul 18 17:09:53.838868 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.838825    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"]
Jul 18 17:09:53.839228 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.839211    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aa122bd-abbf-4700-97bc-bcc417369f45" containerName="maas-api"
Jul 18 17:09:53.839315 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.839231    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aa122bd-abbf-4700-97bc-bcc417369f45" containerName="maas-api"
Jul 18 17:09:53.839315 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.839256    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3525897-7911-4738-bc82-246b7db1befd" containerName="revoke-keys"
Jul 18 17:09:53.839315 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.839265    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3525897-7911-4738-bc82-246b7db1befd" containerName="revoke-keys"
Jul 18 17:09:53.839463 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.839338    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aa122bd-abbf-4700-97bc-bcc417369f45" containerName="maas-api"
Jul 18 17:09:53.839463 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.839351    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3525897-7911-4738-bc82-246b7db1befd" containerName="revoke-keys"
Jul 18 17:09:53.843621 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.843599    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:53.848933 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.848913    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-08db0042-openshift-default-dockercfg-g47l7\""
Jul 18 17:09:53.849581 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.849565    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 18 17:09:53.873328 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.873298    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"]
Jul 18 17:09:53.894387 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:53.894344    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:54.005953 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.005916    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-data\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.005953 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.005954    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-token\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.006168 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.006013    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-podinfo\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.006168 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.006052    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-workload-certs\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.006168 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.006097    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0c70c5-e44c-4165-91c1-70d73ba12870-istiod-ca-cert\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.006168 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.006121    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gshdf\" (UniqueName: \"kubernetes.io/projected/dd0c70c5-e44c-4165-91c1-70d73ba12870-kube-api-access-gshdf\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.006168 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.006158    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-credential-socket\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.006363 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.006177    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-envoy\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.006363 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.006215    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-workload-socket\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.013008 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.012984    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:54.106933 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.106896    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gshdf\" (UniqueName: \"kubernetes.io/projected/dd0c70c5-e44c-4165-91c1-70d73ba12870-kube-api-access-gshdf\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.107368 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.106940    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-credential-socket\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.107368 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.106967    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-envoy\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.107368 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.107008    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-workload-socket\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.107368 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.107084    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-data\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.107368 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.107108    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-token\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.107368 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.107133    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-podinfo\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.107368 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.107160    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-workload-certs\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.107368 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.107217    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0c70c5-e44c-4165-91c1-70d73ba12870-istiod-ca-cert\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.107368 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.107361    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-credential-socket\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.107829 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.107480    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-workload-socket\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.107829 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.107529    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-workload-certs\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.107829 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.107754    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-data\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.107829 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.107828    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0c70c5-e44c-4165-91c1-70d73ba12870-istiod-ca-cert\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.109345 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.109323    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-envoy\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.109629 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.109610    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-podinfo\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.115703 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.115678    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-token\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.116106 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.116075    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gshdf\" (UniqueName: \"kubernetes.io/projected/dd0c70c5-e44c-4165-91c1-70d73ba12870-kube-api-access-gshdf\") pod \"e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") " pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.154038 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.154012    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:54.290603 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.290576    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"]
Jul 18 17:09:54.293421 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:09:54.293391    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0c70c5_e44c_4165_91c1_70d73ba12870.slice/crio-971c49970ba39ae3deb819cc2623d5836a18d4198877f984e9ed0be4b8388eec WatchSource:0}: Error finding container 971c49970ba39ae3deb819cc2623d5836a18d4198877f984e9ed0be4b8388eec: Status 404 returned error can't find the container with id 971c49970ba39ae3deb819cc2623d5836a18d4198877f984e9ed0be4b8388eec
Jul 18 17:09:54.295591 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.295559    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:09:54.295671 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.295625    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:09:54.295671 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:54.295657    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:09:55.034977 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:55.034939    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5" event={"ID":"dd0c70c5-e44c-4165-91c1-70d73ba12870","Type":"ContainerStarted","Data":"acae12996147e1df2922f4defff2da02e3fb193efe9e615086ec66c1e684e697"}
Jul 18 17:09:55.035159 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:55.034995    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5" event={"ID":"dd0c70c5-e44c-4165-91c1-70d73ba12870","Type":"ContainerStarted","Data":"971c49970ba39ae3deb819cc2623d5836a18d4198877f984e9ed0be4b8388eec"}
Jul 18 17:09:55.056204 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:55.056154    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5" podStartSLOduration=2.05613982 podStartE2EDuration="2.05613982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:09:55.054400379 +0000 UTC m=+2013.752231064" watchObservedRunningTime="2026-07-18 17:09:55.05613982 +0000 UTC m=+2013.753970504"
Jul 18 17:09:55.154162 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:55.154125    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:55.155761 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:55.155733    2554 patch_prober.go:28] interesting pod/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 18 17:09:55.155901 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:55.155787    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5" podUID="dd0c70c5-e44c-4165-91c1-70d73ba12870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 18 17:09:56.159125 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.159093    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:56.390008 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.389969    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:56.481024 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.480947    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:56.644446 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.644412    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"]
Jul 18 17:09:56.647647 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.647628    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:09:56.650203 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.650175    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-08db0042-serving-cert\""
Jul 18 17:09:56.650319 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.650209    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 18 17:09:56.650374 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.650333    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 18 17:09:56.651304 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.651284    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2pqsv\""
Jul 18 17:09:56.658047 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.658024    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"]
Jul 18 17:09:56.675479 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.675460    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:56.831245 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.831214    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8ts4\" (UniqueName: \"kubernetes.io/projected/2bb1a207-c98a-4d79-9987-e17542189aa1-kube-api-access-b8ts4\") pod \"maas-api-e2e-derive-08db0042-7b65984699-dw79s\" (UID: \"2bb1a207-c98a-4d79-9987-e17542189aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:09:56.831418 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.831271    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb1a207-c98a-4d79-9987-e17542189aa1-maas-api-tls\") pod \"maas-api-e2e-derive-08db0042-7b65984699-dw79s\" (UID: \"2bb1a207-c98a-4d79-9987-e17542189aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:09:56.831418 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.831324    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb1a207-c98a-4d79-9987-e17542189aa1-openshift-service-ca\") pod \"maas-api-e2e-derive-08db0042-7b65984699-dw79s\" (UID: \"2bb1a207-c98a-4d79-9987-e17542189aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:09:56.932238 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.932206    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb1a207-c98a-4d79-9987-e17542189aa1-maas-api-tls\") pod \"maas-api-e2e-derive-08db0042-7b65984699-dw79s\" (UID: \"2bb1a207-c98a-4d79-9987-e17542189aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:09:56.932394 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.932247    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb1a207-c98a-4d79-9987-e17542189aa1-openshift-service-ca\") pod \"maas-api-e2e-derive-08db0042-7b65984699-dw79s\" (UID: \"2bb1a207-c98a-4d79-9987-e17542189aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:09:56.932394 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.932298    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8ts4\" (UniqueName: \"kubernetes.io/projected/2bb1a207-c98a-4d79-9987-e17542189aa1-kube-api-access-b8ts4\") pod \"maas-api-e2e-derive-08db0042-7b65984699-dw79s\" (UID: \"2bb1a207-c98a-4d79-9987-e17542189aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:09:56.932394 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:09:56.932365    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-serving-cert: secret "maas-api-e2e-derive-08db0042-serving-cert" not found
Jul 18 17:09:56.932496 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:09:56.932442    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bb1a207-c98a-4d79-9987-e17542189aa1-maas-api-tls podName:2bb1a207-c98a-4d79-9987-e17542189aa1 nodeName:}" failed. No retries permitted until 2026-07-18 17:09:57.432421674 +0000 UTC m=+2016.130252353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2bb1a207-c98a-4d79-9987-e17542189aa1-maas-api-tls") pod "maas-api-e2e-derive-08db0042-7b65984699-dw79s" (UID: "2bb1a207-c98a-4d79-9987-e17542189aa1") : secret "maas-api-e2e-derive-08db0042-serving-cert" not found
Jul 18 17:09:56.935841 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.933267    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb1a207-c98a-4d79-9987-e17542189aa1-openshift-service-ca\") pod \"maas-api-e2e-derive-08db0042-7b65984699-dw79s\" (UID: \"2bb1a207-c98a-4d79-9987-e17542189aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:09:56.942728 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:56.942699    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8ts4\" (UniqueName: \"kubernetes.io/projected/2bb1a207-c98a-4d79-9987-e17542189aa1-kube-api-access-b8ts4\") pod \"maas-api-e2e-derive-08db0042-7b65984699-dw79s\" (UID: \"2bb1a207-c98a-4d79-9987-e17542189aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:09:57.044278 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:57.044252    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:57.045265 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:57.045239    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:09:57.125285 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:57.125204    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:57.223791 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:57.223751    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:57.330320 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:57.330293    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:09:57.437794 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:57.437695    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb1a207-c98a-4d79-9987-e17542189aa1-maas-api-tls\") pod \"maas-api-e2e-derive-08db0042-7b65984699-dw79s\" (UID: \"2bb1a207-c98a-4d79-9987-e17542189aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:09:57.440357 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:57.440334    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb1a207-c98a-4d79-9987-e17542189aa1-maas-api-tls\") pod \"maas-api-e2e-derive-08db0042-7b65984699-dw79s\" (UID: \"2bb1a207-c98a-4d79-9987-e17542189aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:09:57.558633 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:57.558596    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:09:57.707874 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:57.707826    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"]
Jul 18 17:09:57.710142 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:09:57.710117    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb1a207_c98a_4d79_9987_e17542189aa1.slice/crio-bd8d28b5e46457075ee496984b4501cdb446b508732f0cf738c98cbf1c0aefe2 WatchSource:0}: Error finding container bd8d28b5e46457075ee496984b4501cdb446b508732f0cf738c98cbf1c0aefe2: Status 404 returned error can't find the container with id bd8d28b5e46457075ee496984b4501cdb446b508732f0cf738c98cbf1c0aefe2
Jul 18 17:09:58.050132 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:58.050053    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s" event={"ID":"2bb1a207-c98a-4d79-9987-e17542189aa1","Type":"ContainerStarted","Data":"bd8d28b5e46457075ee496984b4501cdb446b508732f0cf738c98cbf1c0aefe2"}
Jul 18 17:09:59.056036 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:59.056000    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s" event={"ID":"2bb1a207-c98a-4d79-9987-e17542189aa1","Type":"ContainerStarted","Data":"8324744f5b88a1eb8191c2ac33c85dc4127445545a0e6edb31e6f2cfc0fcc3da"}
Jul 18 17:09:59.073032 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:09:59.072985    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s" podStartSLOduration=2.671941912 podStartE2EDuration="3.072969702s" totalImagesPullingTime="401.02779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:09:59.071985294 +0000 UTC m=+2017.769815992" watchObservedRunningTime="2026-07-18 17:09:59.072969702 +0000 UTC m=+2017.770800386"
Jul 18 17:10:00.060780 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:00.060754    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:10:06.076413 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:06.076380    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:10:08.615913 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:08.615874    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:08.725658 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:08.725619    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:10.117323 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:10.117292    2554 generic.go:356] "Generic (PLEG): container finished" podID="dd0c70c5-e44c-4165-91c1-70d73ba12870" containerID="acae12996147e1df2922f4defff2da02e3fb193efe9e615086ec66c1e684e697" exitCode=0
Jul 18 17:10:10.117750 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:10.117358    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5" event={"ID":"dd0c70c5-e44c-4165-91c1-70d73ba12870","Type":"ContainerDied","Data":"acae12996147e1df2922f4defff2da02e3fb193efe9e615086ec66c1e684e697"}
Jul 18 17:10:10.117796 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:10.117782    2554 scope.go:117] "RemoveContainer" containerID="acae12996147e1df2922f4defff2da02e3fb193efe9e615086ec66c1e684e697"
Jul 18 17:10:10.118777 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:10.118748    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:10:10.118863 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:10.118827    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:10:10.118905 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:10.118885    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:10:11.124082 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:11.124044    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5" event={"ID":"dd0c70c5-e44c-4165-91c1-70d73ba12870","Type":"ContainerStarted","Data":"0461d7b7631da53106b383003fbb0149bd40815978af8d2fce9958ebefffbb12"}
Jul 18 17:10:11.154810 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:11.154781    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:10:11.155704 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:11.155683    2554 patch_prober.go:28] interesting pod/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 18 17:10:11.155798 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:11.155749    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5" podUID="dd0c70c5-e44c-4165-91c1-70d73ba12870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 18 17:10:12.044678 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:12.044641    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:10:12.155213 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:12.155179    2554 patch_prober.go:28] interesting pod/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 18 17:10:12.155665 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:12.155237    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5" podUID="dd0c70c5-e44c-4165-91c1-70d73ba12870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 18 17:10:13.155129 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:13.155094    2554 patch_prober.go:28] interesting pod/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 18 17:10:13.155295 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:13.155148    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5" podUID="dd0c70c5-e44c-4165-91c1-70d73ba12870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 18 17:10:14.155348 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:14.155315    2554 patch_prober.go:28] interesting pod/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 18 17:10:14.155724 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:14.155371    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5" podUID="dd0c70c5-e44c-4165-91c1-70d73ba12870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 18 17:10:15.158324 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:15.158296    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:10:15.159327 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:15.159308    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:10:17.967565 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:17.967532    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:18.071428 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:18.071391    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:21.275573 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:21.275534    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf"]
Jul 18 17:10:21.277903 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:21.277881    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf"
Jul 18 17:10:21.280425 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:21.280402    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7rq8\""
Jul 18 17:10:21.287138 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:21.287114    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf"]
Jul 18 17:10:21.357123 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:21.357074    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f77173b-da5b-4145-afb0-ca3e0c064b62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-08db0042-sk9pf\" (UID: \"8f77173b-da5b-4145-afb0-ca3e0c064b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf"
Jul 18 17:10:21.465042 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:21.459333    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f77173b-da5b-4145-afb0-ca3e0c064b62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-08db0042-sk9pf\" (UID: \"8f77173b-da5b-4145-afb0-ca3e0c064b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf"
Jul 18 17:10:21.465042 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:21.460245    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f77173b-da5b-4145-afb0-ca3e0c064b62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-08db0042-sk9pf\" (UID: \"8f77173b-da5b-4145-afb0-ca3e0c064b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf"
Jul 18 17:10:21.588770 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:21.588733    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf"
Jul 18 17:10:21.720301 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:21.720133    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf"]
Jul 18 17:10:21.722920 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:10:21.722882    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f77173b_da5b_4145_afb0_ca3e0c064b62.slice/crio-591deef57f57b3fe45a133c4cfab393d500f63f7d4d1da0d5a6617fcff6514e1 WatchSource:0}: Error finding container 591deef57f57b3fe45a133c4cfab393d500f63f7d4d1da0d5a6617fcff6514e1: Status 404 returned error can't find the container with id 591deef57f57b3fe45a133c4cfab393d500f63f7d4d1da0d5a6617fcff6514e1
Jul 18 17:10:22.181115 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:22.181032    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf" event={"ID":"8f77173b-da5b-4145-afb0-ca3e0c064b62","Type":"ContainerStarted","Data":"e25776780660312a09b5453acc1ad68a6e5c3ad8fdaaf57bf8a95051abef08a5"}
Jul 18 17:10:22.181115 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:22.181083    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf" event={"ID":"8f77173b-da5b-4145-afb0-ca3e0c064b62","Type":"ContainerStarted","Data":"591deef57f57b3fe45a133c4cfab393d500f63f7d4d1da0d5a6617fcff6514e1"}
Jul 18 17:10:22.197786 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:22.197737    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf" podStartSLOduration=1.197721666 podStartE2EDuration="1.197721666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:10:22.196507484 +0000 UTC m=+2040.894338172" watchObservedRunningTime="2026-07-18 17:10:22.197721666 +0000 UTC m=+2040.895552350"
Jul 18 17:10:23.187607 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:23.187573    2554 generic.go:356] "Generic (PLEG): container finished" podID="8f77173b-da5b-4145-afb0-ca3e0c064b62" containerID="e25776780660312a09b5453acc1ad68a6e5c3ad8fdaaf57bf8a95051abef08a5" exitCode=0
Jul 18 17:10:23.188055 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:23.187663    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf" event={"ID":"8f77173b-da5b-4145-afb0-ca3e0c064b62","Type":"ContainerDied","Data":"e25776780660312a09b5453acc1ad68a6e5c3ad8fdaaf57bf8a95051abef08a5"}
Jul 18 17:10:24.328266 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:24.328243    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf"
Jul 18 17:10:24.385993 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:24.385964    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f77173b-da5b-4145-afb0-ca3e0c064b62-maas-api-service-ca\") pod \"8f77173b-da5b-4145-afb0-ca3e0c064b62\" (UID: \"8f77173b-da5b-4145-afb0-ca3e0c064b62\") "
Jul 18 17:10:24.386294 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:24.386273    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f77173b-da5b-4145-afb0-ca3e0c064b62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f77173b-da5b-4145-afb0-ca3e0c064b62" (UID: "8f77173b-da5b-4145-afb0-ca3e0c064b62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:10:24.487561 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:24.487481    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f77173b-da5b-4145-afb0-ca3e0c064b62-maas-api-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:10:25.197577 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:25.197534    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf" event={"ID":"8f77173b-da5b-4145-afb0-ca3e0c064b62","Type":"ContainerDied","Data":"591deef57f57b3fe45a133c4cfab393d500f63f7d4d1da0d5a6617fcff6514e1"}
Jul 18 17:10:25.197577 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:25.197575    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf"
Jul 18 17:10:25.197838 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:25.197591    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="591deef57f57b3fe45a133c4cfab393d500f63f7d4d1da0d5a6617fcff6514e1"
Jul 18 17:10:31.687022 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:31.686990    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"]
Jul 18 17:10:31.687455 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:31.687193    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s" podUID="2bb1a207-c98a-4d79-9987-e17542189aa1" containerName="maas-api" containerID="cri-o://8324744f5b88a1eb8191c2ac33c85dc4127445545a0e6edb31e6f2cfc0fcc3da" gracePeriod=30
Jul 18 17:10:31.732329 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:31.732302    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:31.936946 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:31.936923    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:10:32.053761 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.053603    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8ts4\" (UniqueName: \"kubernetes.io/projected/2bb1a207-c98a-4d79-9987-e17542189aa1-kube-api-access-b8ts4\") pod \"2bb1a207-c98a-4d79-9987-e17542189aa1\" (UID: \"2bb1a207-c98a-4d79-9987-e17542189aa1\") "
Jul 18 17:10:32.053937 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.053773    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb1a207-c98a-4d79-9987-e17542189aa1-maas-api-tls\") pod \"2bb1a207-c98a-4d79-9987-e17542189aa1\" (UID: \"2bb1a207-c98a-4d79-9987-e17542189aa1\") "
Jul 18 17:10:32.053937 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.053841    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb1a207-c98a-4d79-9987-e17542189aa1-openshift-service-ca\") pod \"2bb1a207-c98a-4d79-9987-e17542189aa1\" (UID: \"2bb1a207-c98a-4d79-9987-e17542189aa1\") "
Jul 18 17:10:32.054227 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.054196    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb1a207-c98a-4d79-9987-e17542189aa1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2bb1a207-c98a-4d79-9987-e17542189aa1" (UID: "2bb1a207-c98a-4d79-9987-e17542189aa1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:10:32.055898 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.055871    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb1a207-c98a-4d79-9987-e17542189aa1-kube-api-access-b8ts4" (OuterVolumeSpecName: "kube-api-access-b8ts4") pod "2bb1a207-c98a-4d79-9987-e17542189aa1" (UID: "2bb1a207-c98a-4d79-9987-e17542189aa1"). InnerVolumeSpecName "kube-api-access-b8ts4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:10:32.055984 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.055913    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb1a207-c98a-4d79-9987-e17542189aa1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2bb1a207-c98a-4d79-9987-e17542189aa1" (UID: "2bb1a207-c98a-4d79-9987-e17542189aa1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:10:32.154646 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.154612    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8ts4\" (UniqueName: \"kubernetes.io/projected/2bb1a207-c98a-4d79-9987-e17542189aa1-kube-api-access-b8ts4\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:10:32.154646 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.154638    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bb1a207-c98a-4d79-9987-e17542189aa1-maas-api-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:10:32.154646 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.154648    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb1a207-c98a-4d79-9987-e17542189aa1-openshift-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:10:32.232449 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.232414    2554 generic.go:356] "Generic (PLEG): container finished" podID="2bb1a207-c98a-4d79-9987-e17542189aa1" containerID="8324744f5b88a1eb8191c2ac33c85dc4127445545a0e6edb31e6f2cfc0fcc3da" exitCode=0
Jul 18 17:10:32.232592 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.232526    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"
Jul 18 17:10:32.232592 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.232536    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s" event={"ID":"2bb1a207-c98a-4d79-9987-e17542189aa1","Type":"ContainerDied","Data":"8324744f5b88a1eb8191c2ac33c85dc4127445545a0e6edb31e6f2cfc0fcc3da"}
Jul 18 17:10:32.232682 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.232612    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s" event={"ID":"2bb1a207-c98a-4d79-9987-e17542189aa1","Type":"ContainerDied","Data":"bd8d28b5e46457075ee496984b4501cdb446b508732f0cf738c98cbf1c0aefe2"}
Jul 18 17:10:32.232682 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.232647    2554 scope.go:117] "RemoveContainer" containerID="8324744f5b88a1eb8191c2ac33c85dc4127445545a0e6edb31e6f2cfc0fcc3da"
Jul 18 17:10:32.242562 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.242544    2554 scope.go:117] "RemoveContainer" containerID="8324744f5b88a1eb8191c2ac33c85dc4127445545a0e6edb31e6f2cfc0fcc3da"
Jul 18 17:10:32.242811 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:10:32.242793    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8324744f5b88a1eb8191c2ac33c85dc4127445545a0e6edb31e6f2cfc0fcc3da\": container with ID starting with 8324744f5b88a1eb8191c2ac33c85dc4127445545a0e6edb31e6f2cfc0fcc3da not found: ID does not exist" containerID="8324744f5b88a1eb8191c2ac33c85dc4127445545a0e6edb31e6f2cfc0fcc3da"
Jul 18 17:10:32.242889 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.242821    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8324744f5b88a1eb8191c2ac33c85dc4127445545a0e6edb31e6f2cfc0fcc3da"} err="failed to get container status \"8324744f5b88a1eb8191c2ac33c85dc4127445545a0e6edb31e6f2cfc0fcc3da\": rpc error: code = NotFound desc = could not find container \"8324744f5b88a1eb8191c2ac33c85dc4127445545a0e6edb31e6f2cfc0fcc3da\": container with ID starting with 8324744f5b88a1eb8191c2ac33c85dc4127445545a0e6edb31e6f2cfc0fcc3da not found: ID does not exist"
Jul 18 17:10:32.256615 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.256593    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"]
Jul 18 17:10:32.259575 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:32.259539    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-08db0042-7b65984699-dw79s"]
Jul 18 17:10:33.765789 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:33.765743    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bb1a207-c98a-4d79-9987-e17542189aa1" path="/var/lib/kubelet/pods/2bb1a207-c98a-4d79-9987-e17542189aa1/volumes"
Jul 18 17:10:36.348052 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:36.348014    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:36.352112 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:36.352091    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf"]
Jul 18 17:10:36.354435 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:36.354417    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-08db0042-sk9pf"]
Jul 18 17:10:36.492770 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:36.492736    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"]
Jul 18 17:10:36.493063 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:36.493037    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5" podUID="dd0c70c5-e44c-4165-91c1-70d73ba12870" containerName="istio-proxy" containerID="cri-o://0461d7b7631da53106b383003fbb0149bd40815978af8d2fce9958ebefffbb12" gracePeriod=30
Jul 18 17:10:36.518403 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:36.518374    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:36.744871 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:36.744751    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:37.220891 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:37.220837    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:37.319704 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:37.319665    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:37.420119 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:37.420086    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:37.766505 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:37.766462    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f77173b-da5b-4145-afb0-ca3e0c064b62" path="/var/lib/kubelet/pods/8f77173b-da5b-4145-afb0-ca3e0c064b62/volumes"
Jul 18 17:10:38.788551 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:38.788520    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:41.730233 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.730209    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:10:41.838045 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.838019    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-envoy\") pod \"dd0c70c5-e44c-4165-91c1-70d73ba12870\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") "
Jul 18 17:10:41.838205 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.838058    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-workload-certs\") pod \"dd0c70c5-e44c-4165-91c1-70d73ba12870\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") "
Jul 18 17:10:41.838205 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.838102    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0c70c5-e44c-4165-91c1-70d73ba12870-istiod-ca-cert\") pod \"dd0c70c5-e44c-4165-91c1-70d73ba12870\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") "
Jul 18 17:10:41.838205 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.838123    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-podinfo\") pod \"dd0c70c5-e44c-4165-91c1-70d73ba12870\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") "
Jul 18 17:10:41.838205 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.838140    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-data\") pod \"dd0c70c5-e44c-4165-91c1-70d73ba12870\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") "
Jul 18 17:10:41.838205 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.838166    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-workload-socket\") pod \"dd0c70c5-e44c-4165-91c1-70d73ba12870\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") "
Jul 18 17:10:41.838205 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.838195    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-token\") pod \"dd0c70c5-e44c-4165-91c1-70d73ba12870\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") "
Jul 18 17:10:41.838478 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.838241    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-credential-socket\") pod \"dd0c70c5-e44c-4165-91c1-70d73ba12870\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") "
Jul 18 17:10:41.838478 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.838280    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gshdf\" (UniqueName: \"kubernetes.io/projected/dd0c70c5-e44c-4165-91c1-70d73ba12870-kube-api-access-gshdf\") pod \"dd0c70c5-e44c-4165-91c1-70d73ba12870\" (UID: \"dd0c70c5-e44c-4165-91c1-70d73ba12870\") "
Jul 18 17:10:41.838588 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.838481    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dd0c70c5-e44c-4165-91c1-70d73ba12870" (UID: "dd0c70c5-e44c-4165-91c1-70d73ba12870"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:10:41.838588 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.838543    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-workload-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:10:41.838588 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.838545    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0c70c5-e44c-4165-91c1-70d73ba12870-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dd0c70c5-e44c-4165-91c1-70d73ba12870" (UID: "dd0c70c5-e44c-4165-91c1-70d73ba12870"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:10:41.838738 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.838514    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dd0c70c5-e44c-4165-91c1-70d73ba12870" (UID: "dd0c70c5-e44c-4165-91c1-70d73ba12870"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:10:41.838738 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.838681    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dd0c70c5-e44c-4165-91c1-70d73ba12870" (UID: "dd0c70c5-e44c-4165-91c1-70d73ba12870"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:10:41.839261 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.839236    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-data" (OuterVolumeSpecName: "istio-data") pod "dd0c70c5-e44c-4165-91c1-70d73ba12870" (UID: "dd0c70c5-e44c-4165-91c1-70d73ba12870"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:10:41.840675 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.840655    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0c70c5-e44c-4165-91c1-70d73ba12870-kube-api-access-gshdf" (OuterVolumeSpecName: "kube-api-access-gshdf") pod "dd0c70c5-e44c-4165-91c1-70d73ba12870" (UID: "dd0c70c5-e44c-4165-91c1-70d73ba12870"). InnerVolumeSpecName "kube-api-access-gshdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:10:41.840675 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.840648    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-token" (OuterVolumeSpecName: "istio-token") pod "dd0c70c5-e44c-4165-91c1-70d73ba12870" (UID: "dd0c70c5-e44c-4165-91c1-70d73ba12870"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:10:41.840864 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.840828    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dd0c70c5-e44c-4165-91c1-70d73ba12870" (UID: "dd0c70c5-e44c-4165-91c1-70d73ba12870"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 17:10:41.840965 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.840948    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dd0c70c5-e44c-4165-91c1-70d73ba12870" (UID: "dd0c70c5-e44c-4165-91c1-70d73ba12870"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:10:41.939831 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.939800    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-credential-socket\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:10:41.939831 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.939824    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gshdf\" (UniqueName: \"kubernetes.io/projected/dd0c70c5-e44c-4165-91c1-70d73ba12870-kube-api-access-gshdf\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:10:41.939831 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.939834    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-envoy\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:10:41.940053 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.939866    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd0c70c5-e44c-4165-91c1-70d73ba12870-istiod-ca-cert\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:10:41.940053 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.939880    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-podinfo\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:10:41.940053 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.939891    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-data\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:10:41.940053 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.939903    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd0c70c5-e44c-4165-91c1-70d73ba12870-workload-socket\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:10:41.940053 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:41.939915    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd0c70c5-e44c-4165-91c1-70d73ba12870-istio-token\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:10:42.279447 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:42.279364    2554 generic.go:356] "Generic (PLEG): container finished" podID="dd0c70c5-e44c-4165-91c1-70d73ba12870" containerID="0461d7b7631da53106b383003fbb0149bd40815978af8d2fce9958ebefffbb12" exitCode=0
Jul 18 17:10:42.279447 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:42.279405    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5" event={"ID":"dd0c70c5-e44c-4165-91c1-70d73ba12870","Type":"ContainerDied","Data":"0461d7b7631da53106b383003fbb0149bd40815978af8d2fce9958ebefffbb12"}
Jul 18 17:10:42.279447 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:42.279414    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"
Jul 18 17:10:42.279447 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:42.279444    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5" event={"ID":"dd0c70c5-e44c-4165-91c1-70d73ba12870","Type":"ContainerDied","Data":"971c49970ba39ae3deb819cc2623d5836a18d4198877f984e9ed0be4b8388eec"}
Jul 18 17:10:42.279715 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:42.279465    2554 scope.go:117] "RemoveContainer" containerID="0461d7b7631da53106b383003fbb0149bd40815978af8d2fce9958ebefffbb12"
Jul 18 17:10:42.288469 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:42.288453    2554 scope.go:117] "RemoveContainer" containerID="acae12996147e1df2922f4defff2da02e3fb193efe9e615086ec66c1e684e697"
Jul 18 17:10:42.296533 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:42.296511    2554 scope.go:117] "RemoveContainer" containerID="0461d7b7631da53106b383003fbb0149bd40815978af8d2fce9958ebefffbb12"
Jul 18 17:10:42.296796 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:10:42.296774    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0461d7b7631da53106b383003fbb0149bd40815978af8d2fce9958ebefffbb12\": container with ID starting with 0461d7b7631da53106b383003fbb0149bd40815978af8d2fce9958ebefffbb12 not found: ID does not exist" containerID="0461d7b7631da53106b383003fbb0149bd40815978af8d2fce9958ebefffbb12"
Jul 18 17:10:42.296899 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:42.296804    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0461d7b7631da53106b383003fbb0149bd40815978af8d2fce9958ebefffbb12"} err="failed to get container status \"0461d7b7631da53106b383003fbb0149bd40815978af8d2fce9958ebefffbb12\": rpc error: code = NotFound desc = could not find container \"0461d7b7631da53106b383003fbb0149bd40815978af8d2fce9958ebefffbb12\": container with ID starting with 0461d7b7631da53106b383003fbb0149bd40815978af8d2fce9958ebefffbb12 not found: ID does not exist"
Jul 18 17:10:42.296899 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:42.296819    2554 scope.go:117] "RemoveContainer" containerID="acae12996147e1df2922f4defff2da02e3fb193efe9e615086ec66c1e684e697"
Jul 18 17:10:42.297100 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:10:42.297083    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acae12996147e1df2922f4defff2da02e3fb193efe9e615086ec66c1e684e697\": container with ID starting with acae12996147e1df2922f4defff2da02e3fb193efe9e615086ec66c1e684e697 not found: ID does not exist" containerID="acae12996147e1df2922f4defff2da02e3fb193efe9e615086ec66c1e684e697"
Jul 18 17:10:42.297138 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:42.297107    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acae12996147e1df2922f4defff2da02e3fb193efe9e615086ec66c1e684e697"} err="failed to get container status \"acae12996147e1df2922f4defff2da02e3fb193efe9e615086ec66c1e684e697\": rpc error: code = NotFound desc = could not find container \"acae12996147e1df2922f4defff2da02e3fb193efe9e615086ec66c1e684e697\": container with ID starting with acae12996147e1df2922f4defff2da02e3fb193efe9e615086ec66c1e684e697 not found: ID does not exist"
Jul 18 17:10:42.302388 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:42.302367    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"]
Jul 18 17:10:42.310743 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:42.308896    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-08db0042-openshift-default-68959b95f5-fbhj5"]
Jul 18 17:10:43.766142 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:43.766061    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd0c70c5-e44c-4165-91c1-70d73ba12870" path="/var/lib/kubelet/pods/dd0c70c5-e44c-4165-91c1-70d73ba12870/volumes"
Jul 18 17:10:44.497457 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:44.497421    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:44.556642 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:44.556609    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:44.629866 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:44.629810    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:44.728986 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:44.728949    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:44.829343 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:44.829313    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:10:45.192880 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:10:45.192771    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:11.151023 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:11.150991    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:17.244507 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:17.244460    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:17.347685 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:17.347640    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:17.443590 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:17.443553    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:33.310471 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:33.310440    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:38.866463 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:38.866423    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:38.931499 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:38.931464    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:39.013402 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:39.013374    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:39.141027 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:39.140946    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:40.070718 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:40.070684    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:49.627666 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:49.627634    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:50.417046 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:50.417011    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:57.600343 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:57.600312    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:57.706113 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:57.706082    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:57.801322 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:57.801293    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:57.901905 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:57.901823    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:58.645376 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:58.645344    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:58.742524 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:58.742492    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:11:58.842796 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:11:58.842754    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:00.439155 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:00.439121    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:06.400782 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:06.400745    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:06.461941 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:06.461912    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:07.051357 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:07.051325    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:12.838543 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:12.838457    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:12.938921 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:12.938883    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:13.173486 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.173406    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:13.380614 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.380581    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"]
Jul 18 17:12:13.380947 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.380931    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f77173b-da5b-4145-afb0-ca3e0c064b62" containerName="revoke-keys"
Jul 18 17:12:13.380947 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.380947    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f77173b-da5b-4145-afb0-ca3e0c064b62" containerName="revoke-keys"
Jul 18 17:12:13.381095 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.380958    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bb1a207-c98a-4d79-9987-e17542189aa1" containerName="maas-api"
Jul 18 17:12:13.381095 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.380964    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bb1a207-c98a-4d79-9987-e17542189aa1" containerName="maas-api"
Jul 18 17:12:13.381095 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.380983    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd0c70c5-e44c-4165-91c1-70d73ba12870" containerName="istio-proxy"
Jul 18 17:12:13.381095 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.380988    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd0c70c5-e44c-4165-91c1-70d73ba12870" containerName="istio-proxy"
Jul 18 17:12:13.381095 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.380996    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd0c70c5-e44c-4165-91c1-70d73ba12870" containerName="istio-proxy"
Jul 18 17:12:13.381095 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.381001    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd0c70c5-e44c-4165-91c1-70d73ba12870" containerName="istio-proxy"
Jul 18 17:12:13.381095 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.381052    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f77173b-da5b-4145-afb0-ca3e0c064b62" containerName="revoke-keys"
Jul 18 17:12:13.381095 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.381060    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd0c70c5-e44c-4165-91c1-70d73ba12870" containerName="istio-proxy"
Jul 18 17:12:13.381095 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.381065    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd0c70c5-e44c-4165-91c1-70d73ba12870" containerName="istio-proxy"
Jul 18 17:12:13.381095 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.381072    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bb1a207-c98a-4d79-9987-e17542189aa1" containerName="maas-api"
Jul 18 17:12:13.383507 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.383488    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.386008 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.385985    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 18 17:12:13.386113 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.386078    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-47d19a6a-openshift-default-dockercfg-9htmr\""
Jul 18 17:12:13.407521 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.407496    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"]
Jul 18 17:12:13.410020 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.409992    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:13.513039 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.512946    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-workload-socket\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.513039 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.512986    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-credential-socket\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.513039 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.513006    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d12222-f3c4-41b1-8fca-53b6083a57e3-istiod-ca-cert\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.513503 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.513090    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-workload-certs\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.513503 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.513132    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-envoy\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.513503 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.513158    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-data\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.513503 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.513222    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg66j\" (UniqueName: \"kubernetes.io/projected/20d12222-f3c4-41b1-8fca-53b6083a57e3-kube-api-access-mg66j\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.513503 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.513263    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-podinfo\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.513503 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.513341    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-token\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.613966 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.613929    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-token\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.613966 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.613973    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-workload-socket\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.614159 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.613991    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-credential-socket\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.614159 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.614007    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d12222-f3c4-41b1-8fca-53b6083a57e3-istiod-ca-cert\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.614159 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.614131    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-workload-certs\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.614304 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.614170    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-envoy\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.614304 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.614198    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-data\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.614304 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.614261    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg66j\" (UniqueName: \"kubernetes.io/projected/20d12222-f3c4-41b1-8fca-53b6083a57e3-kube-api-access-mg66j\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.614304 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.614300    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-podinfo\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.614623 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.614431    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-workload-socket\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.614623 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.614539    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-workload-certs\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.614921 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.614896    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-data\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.615077 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.614941    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d12222-f3c4-41b1-8fca-53b6083a57e3-istiod-ca-cert\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.615265 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.615246    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-credential-socket\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.616747 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.616721    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-envoy\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.616980 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.616960    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-podinfo\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.622906 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.622881    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-token\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.623075 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.623061    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg66j\" (UniqueName: \"kubernetes.io/projected/20d12222-f3c4-41b1-8fca-53b6083a57e3-kube-api-access-mg66j\") pod \"e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") " pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.694188 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.694147    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:13.771719 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.771628    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:13.851275 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.851248    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"]
Jul 18 17:12:13.854525 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:12:13.854495    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d12222_f3c4_41b1_8fca_53b6083a57e3.slice/crio-0768f608f6327c90de65de024191b2de3d73a9c7d746b82662e4e740b4a37b67 WatchSource:0}: Error finding container 0768f608f6327c90de65de024191b2de3d73a9c7d746b82662e4e740b4a37b67: Status 404 returned error can't find the container with id 0768f608f6327c90de65de024191b2de3d73a9c7d746b82662e4e740b4a37b67
Jul 18 17:12:13.856805 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.856773    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:12:13.856932 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.856873    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:12:13.856932 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:13.856925    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:12:14.705576 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:14.705539    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8" event={"ID":"20d12222-f3c4-41b1-8fca-53b6083a57e3","Type":"ContainerStarted","Data":"f5acb5dd8a20f863a62e16a9a058dd060aa993e2ce2dea2966ecfa4a1d2014c2"}
Jul 18 17:12:14.705792 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:14.705610    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8" event={"ID":"20d12222-f3c4-41b1-8fca-53b6083a57e3","Type":"ContainerStarted","Data":"0768f608f6327c90de65de024191b2de3d73a9c7d746b82662e4e740b4a37b67"}
Jul 18 17:12:14.726318 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:14.726256    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8" podStartSLOduration=1.7262390779999999 podStartE2EDuration="1.726239078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:12:14.724066689 +0000 UTC m=+2153.421897388" watchObservedRunningTime="2026-07-18 17:12:14.726239078 +0000 UTC m=+2153.424069762"
Jul 18 17:12:15.694611 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:15.694569    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:15.700298 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:15.700262    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:15.710993 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:15.710967    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:15.711718 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:15.711697    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:16.598129 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.598094    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:16.678059 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.678029    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"]
Jul 18 17:12:16.682019 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.681991    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"
Jul 18 17:12:16.682716 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.682690    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:16.684705 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.684684    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2pqsv\""
Jul 18 17:12:16.684819 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.684724    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-47d19a6a-serving-cert\""
Jul 18 17:12:16.684819 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.684747    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 18 17:12:16.684819 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.684694    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 18 17:12:16.690359 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.690334    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"]
Jul 18 17:12:16.743292 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.743266    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-maas-api-tls\") pod \"maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb\" (UID: \"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"
Jul 18 17:12:16.743632 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.743317    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlb9n\" (UniqueName: \"kubernetes.io/projected/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-kube-api-access-nlb9n\") pod \"maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb\" (UID: \"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"
Jul 18 17:12:16.743632 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.743349    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-openshift-service-ca\") pod \"maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb\" (UID: \"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"
Jul 18 17:12:16.745460 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.745439    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:16.844202 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.844171    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlb9n\" (UniqueName: \"kubernetes.io/projected/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-kube-api-access-nlb9n\") pod \"maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb\" (UID: \"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"
Jul 18 17:12:16.844359 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.844213    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-openshift-service-ca\") pod \"maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb\" (UID: \"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"
Jul 18 17:12:16.844359 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.844339    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-maas-api-tls\") pod \"maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb\" (UID: \"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"
Jul 18 17:12:16.845040 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.845016    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-openshift-service-ca\") pod \"maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb\" (UID: \"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"
Jul 18 17:12:16.846890 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.846830    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:16.847042 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.847026    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-maas-api-tls\") pod \"maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb\" (UID: \"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"
Jul 18 17:12:16.854407 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.854356    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlb9n\" (UniqueName: \"kubernetes.io/projected/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-kube-api-access-nlb9n\") pod \"maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb\" (UID: \"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"
Jul 18 17:12:16.993574 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:16.993538    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"
Jul 18 17:12:17.148593 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:17.148492    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"]
Jul 18 17:12:17.151165 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:12:17.151137    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc079e1e6_e0e3_4a41_b4ac_6a32dffaaae4.slice/crio-ef00ad38708adb47cb7b9feb49599f632d2004c9c660145c6b9aab7063dff5de WatchSource:0}: Error finding container ef00ad38708adb47cb7b9feb49599f632d2004c9c660145c6b9aab7063dff5de: Status 404 returned error can't find the container with id ef00ad38708adb47cb7b9feb49599f632d2004c9c660145c6b9aab7063dff5de
Jul 18 17:12:17.407741 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:17.407662    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:17.509711 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:17.509684    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:17.721912 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:17.721809    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb" event={"ID":"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4","Type":"ContainerStarted","Data":"5b8e65c9da2994abbff35b7ac3a1353a6c87fb36e2886cb453709877dd1c5021"}
Jul 18 17:12:17.722125 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:17.722101    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"
Jul 18 17:12:17.722189 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:17.722146    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb" event={"ID":"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4","Type":"ContainerStarted","Data":"ef00ad38708adb47cb7b9feb49599f632d2004c9c660145c6b9aab7063dff5de"}
Jul 18 17:12:17.743109 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:17.743061    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb" podStartSLOduration=1.38726011 podStartE2EDuration="1.743045925s" totalImagesPullingTime="355.785815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:12:17.740547148 +0000 UTC m=+2156.438377831" watchObservedRunningTime="2026-07-18 17:12:17.743045925 +0000 UTC m=+2156.440876612"
Jul 18 17:12:21.074296 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:21.074221    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:21.698913 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:21.698875    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:21.799398 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:21.799366    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:22.092134 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:22.092093    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:24.738744 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:24.738714    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"
Jul 18 17:12:24.929226 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:24.929191    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:25.029594 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:25.029499    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:26.766024 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:26.765995    2554 generic.go:356] "Generic (PLEG): container finished" podID="20d12222-f3c4-41b1-8fca-53b6083a57e3" containerID="f5acb5dd8a20f863a62e16a9a058dd060aa993e2ce2dea2966ecfa4a1d2014c2" exitCode=0
Jul 18 17:12:26.766355 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:26.766074    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8" event={"ID":"20d12222-f3c4-41b1-8fca-53b6083a57e3","Type":"ContainerDied","Data":"f5acb5dd8a20f863a62e16a9a058dd060aa993e2ce2dea2966ecfa4a1d2014c2"}
Jul 18 17:12:26.766464 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:26.766447    2554 scope.go:117] "RemoveContainer" containerID="f5acb5dd8a20f863a62e16a9a058dd060aa993e2ce2dea2966ecfa4a1d2014c2"
Jul 18 17:12:26.767468 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:26.767441    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:12:26.767556 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:26.767505    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:12:26.767556 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:26.767536    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:12:27.781644 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:27.780929    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8" event={"ID":"20d12222-f3c4-41b1-8fca-53b6083a57e3","Type":"ContainerStarted","Data":"21eec874f9162e60cf0a2c470ae43187b54c17065ca5c5de4654c5e884165093"}
Jul 18 17:12:28.695389 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:28.695346    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:28.696511 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:28.696485    2554 patch_prober.go:28] interesting pod/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 18 17:12:28.696631 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:28.696536    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8" podUID="20d12222-f3c4-41b1-8fca-53b6083a57e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 18 17:12:28.799613 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:28.799583    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:28.921089 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:28.921049    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:29.192537 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:29.192499    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn"]
Jul 18 17:12:29.195756 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:29.195740    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn"
Jul 18 17:12:29.199418 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:29.199400    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7rq8\""
Jul 18 17:12:29.209480 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:29.209460    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn"]
Jul 18 17:12:29.360296 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:29.360263    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cedb3f78-1ded-4614-8f2e-c83fc76ddce8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn\" (UID: \"cedb3f78-1ded-4614-8f2e-c83fc76ddce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn"
Jul 18 17:12:29.461779 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:29.461689    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cedb3f78-1ded-4614-8f2e-c83fc76ddce8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn\" (UID: \"cedb3f78-1ded-4614-8f2e-c83fc76ddce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn"
Jul 18 17:12:29.462395 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:29.462368    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cedb3f78-1ded-4614-8f2e-c83fc76ddce8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn\" (UID: \"cedb3f78-1ded-4614-8f2e-c83fc76ddce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn"
Jul 18 17:12:29.505398 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:29.505362    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn"
Jul 18 17:12:29.685322 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:29.685285    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn"]
Jul 18 17:12:29.694647 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:29.694625    2554 patch_prober.go:28] interesting pod/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 18 17:12:29.694734 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:29.694675    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8" podUID="20d12222-f3c4-41b1-8fca-53b6083a57e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 18 17:12:29.794654 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:29.794619    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn" event={"ID":"cedb3f78-1ded-4614-8f2e-c83fc76ddce8","Type":"ContainerStarted","Data":"0bd55bbcf133b1dac11b2e36cf7c23899db79c5660a743f9c57de0832c0153c8"}
Jul 18 17:12:30.695300 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:30.695267    2554 patch_prober.go:28] interesting pod/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 18 17:12:30.695595 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:30.695325    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8" podUID="20d12222-f3c4-41b1-8fca-53b6083a57e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 18 17:12:30.710835 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:30.710810    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:30.801131 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:30.801090    2554 generic.go:356] "Generic (PLEG): container finished" podID="cedb3f78-1ded-4614-8f2e-c83fc76ddce8" containerID="28ab753e693dbd22ef3167f13661fbd522c6f93434e6e8243c2e0560514a1dbe" exitCode=0
Jul 18 17:12:30.801305 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:30.801179    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn" event={"ID":"cedb3f78-1ded-4614-8f2e-c83fc76ddce8","Type":"ContainerDied","Data":"28ab753e693dbd22ef3167f13661fbd522c6f93434e6e8243c2e0560514a1dbe"}
Jul 18 17:12:31.698630 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:31.698598    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:31.699743 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:31.699719    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:31.935441 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:31.935419    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn"
Jul 18 17:12:32.108868 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:32.108810    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cedb3f78-1ded-4614-8f2e-c83fc76ddce8-maas-api-service-ca\") pod \"cedb3f78-1ded-4614-8f2e-c83fc76ddce8\" (UID: \"cedb3f78-1ded-4614-8f2e-c83fc76ddce8\") "
Jul 18 17:12:32.109172 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:32.109150    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cedb3f78-1ded-4614-8f2e-c83fc76ddce8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cedb3f78-1ded-4614-8f2e-c83fc76ddce8" (UID: "cedb3f78-1ded-4614-8f2e-c83fc76ddce8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:12:32.209902 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:32.209872    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cedb3f78-1ded-4614-8f2e-c83fc76ddce8-maas-api-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:12:32.812010 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:32.811975    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn" event={"ID":"cedb3f78-1ded-4614-8f2e-c83fc76ddce8","Type":"ContainerDied","Data":"0bd55bbcf133b1dac11b2e36cf7c23899db79c5660a743f9c57de0832c0153c8"}
Jul 18 17:12:32.812010 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:32.811993    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn"
Jul 18 17:12:32.812455 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:32.812029    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bd55bbcf133b1dac11b2e36cf7c23899db79c5660a743f9c57de0832c0153c8"
Jul 18 17:12:39.575301 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.575269    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"]
Jul 18 17:12:39.575746 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.575467    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb" podUID="c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4" containerName="maas-api" containerID="cri-o://5b8e65c9da2994abbff35b7ac3a1353a6c87fb36e2886cb453709877dd1c5021" gracePeriod=30
Jul 18 17:12:39.601873 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.601821    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:39.733253 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.733213    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb" podUID="c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.42:8443/health\": dial tcp 10.133.0.42:8443: connect: connection refused"
Jul 18 17:12:39.820401 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.820380    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"
Jul 18 17:12:39.847224 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.847195    2554 generic.go:356] "Generic (PLEG): container finished" podID="c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4" containerID="5b8e65c9da2994abbff35b7ac3a1353a6c87fb36e2886cb453709877dd1c5021" exitCode=0
Jul 18 17:12:39.847380 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.847245    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"
Jul 18 17:12:39.847380 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.847269    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb" event={"ID":"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4","Type":"ContainerDied","Data":"5b8e65c9da2994abbff35b7ac3a1353a6c87fb36e2886cb453709877dd1c5021"}
Jul 18 17:12:39.847380 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.847323    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb" event={"ID":"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4","Type":"ContainerDied","Data":"ef00ad38708adb47cb7b9feb49599f632d2004c9c660145c6b9aab7063dff5de"}
Jul 18 17:12:39.847380 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.847347    2554 scope.go:117] "RemoveContainer" containerID="5b8e65c9da2994abbff35b7ac3a1353a6c87fb36e2886cb453709877dd1c5021"
Jul 18 17:12:39.855767 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.855747    2554 scope.go:117] "RemoveContainer" containerID="5b8e65c9da2994abbff35b7ac3a1353a6c87fb36e2886cb453709877dd1c5021"
Jul 18 17:12:39.856061 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:12:39.856041    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8e65c9da2994abbff35b7ac3a1353a6c87fb36e2886cb453709877dd1c5021\": container with ID starting with 5b8e65c9da2994abbff35b7ac3a1353a6c87fb36e2886cb453709877dd1c5021 not found: ID does not exist" containerID="5b8e65c9da2994abbff35b7ac3a1353a6c87fb36e2886cb453709877dd1c5021"
Jul 18 17:12:39.856143 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.856068    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8e65c9da2994abbff35b7ac3a1353a6c87fb36e2886cb453709877dd1c5021"} err="failed to get container status \"5b8e65c9da2994abbff35b7ac3a1353a6c87fb36e2886cb453709877dd1c5021\": rpc error: code = NotFound desc = could not find container \"5b8e65c9da2994abbff35b7ac3a1353a6c87fb36e2886cb453709877dd1c5021\": container with ID starting with 5b8e65c9da2994abbff35b7ac3a1353a6c87fb36e2886cb453709877dd1c5021 not found: ID does not exist"
Jul 18 17:12:39.864513 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.864487    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-openshift-service-ca\") pod \"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4\" (UID: \"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4\") "
Jul 18 17:12:39.864589 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.864526    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-maas-api-tls\") pod \"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4\" (UID: \"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4\") "
Jul 18 17:12:39.864589 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.864559    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlb9n\" (UniqueName: \"kubernetes.io/projected/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-kube-api-access-nlb9n\") pod \"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4\" (UID: \"c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4\") "
Jul 18 17:12:39.864916 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.864893    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4" (UID: "c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:12:39.866711 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.866690    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4" (UID: "c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:12:39.866794 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.866737    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-kube-api-access-nlb9n" (OuterVolumeSpecName: "kube-api-access-nlb9n") pod "c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4" (UID: "c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4"). InnerVolumeSpecName "kube-api-access-nlb9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:12:39.966199 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.966163    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-openshift-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:12:39.966199 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.966194    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-maas-api-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:12:39.966199 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:39.966204    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlb9n\" (UniqueName: \"kubernetes.io/projected/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4-kube-api-access-nlb9n\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:12:40.170672 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:40.170632    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"]
Jul 18 17:12:40.172689 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:40.172666    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-47d19a6a-5fcdc8768d-mbpdb"]
Jul 18 17:12:41.766335 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:41.766301    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4" path="/var/lib/kubelet/pods/c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4/volumes"
Jul 18 17:12:44.249531 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:44.249499    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn"]
Jul 18 17:12:44.252047 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:44.252023    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47d19a6a-qn2cn"]
Jul 18 17:12:44.252566 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:44.252545    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:44.718707 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:44.718656    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:45.766477 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:45.766444    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cedb3f78-1ded-4614-8f2e-c83fc76ddce8" path="/var/lib/kubelet/pods/cedb3f78-1ded-4614-8f2e-c83fc76ddce8/volumes"
Jul 18 17:12:50.392840 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:50.392806    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:50.493821 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:50.493790    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:50.627751 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:50.627710    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"]
Jul 18 17:12:50.628052 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:50.628006    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8" podUID="20d12222-f3c4-41b1-8fca-53b6083a57e3" containerName="istio-proxy" containerID="cri-o://21eec874f9162e60cf0a2c470ae43187b54c17065ca5c5de4654c5e884165093" gracePeriod=30
Jul 18 17:12:50.649420 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:50.649357    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:50.877744 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:50.877710    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:50.990261 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:50.990189    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:51.093701 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:51.093667    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:51.206863 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:51.206820    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:55.867864 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.867818    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:55.906034 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.905958    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-data\") pod \"20d12222-f3c4-41b1-8fca-53b6083a57e3\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") "
Jul 18 17:12:55.906034 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.905988    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-credential-socket\") pod \"20d12222-f3c4-41b1-8fca-53b6083a57e3\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") "
Jul 18 17:12:55.906034 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.906027    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-envoy\") pod \"20d12222-f3c4-41b1-8fca-53b6083a57e3\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") "
Jul 18 17:12:55.906273 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.906080    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-workload-certs\") pod \"20d12222-f3c4-41b1-8fca-53b6083a57e3\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") "
Jul 18 17:12:55.906273 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.906099    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-podinfo\") pod \"20d12222-f3c4-41b1-8fca-53b6083a57e3\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") "
Jul 18 17:12:55.906273 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.906114    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-token\") pod \"20d12222-f3c4-41b1-8fca-53b6083a57e3\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") "
Jul 18 17:12:55.906273 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.906136    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg66j\" (UniqueName: \"kubernetes.io/projected/20d12222-f3c4-41b1-8fca-53b6083a57e3-kube-api-access-mg66j\") pod \"20d12222-f3c4-41b1-8fca-53b6083a57e3\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") "
Jul 18 17:12:55.906273 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.906185    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-workload-socket\") pod \"20d12222-f3c4-41b1-8fca-53b6083a57e3\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") "
Jul 18 17:12:55.906530 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.906293    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d12222-f3c4-41b1-8fca-53b6083a57e3-istiod-ca-cert\") pod \"20d12222-f3c4-41b1-8fca-53b6083a57e3\" (UID: \"20d12222-f3c4-41b1-8fca-53b6083a57e3\") "
Jul 18 17:12:55.906530 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.906346    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "20d12222-f3c4-41b1-8fca-53b6083a57e3" (UID: "20d12222-f3c4-41b1-8fca-53b6083a57e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:12:55.906530 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.906365    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "20d12222-f3c4-41b1-8fca-53b6083a57e3" (UID: "20d12222-f3c4-41b1-8fca-53b6083a57e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:12:55.906530 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.906519    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "20d12222-f3c4-41b1-8fca-53b6083a57e3" (UID: "20d12222-f3c4-41b1-8fca-53b6083a57e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:12:55.906748 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.906710    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d12222-f3c4-41b1-8fca-53b6083a57e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "20d12222-f3c4-41b1-8fca-53b6083a57e3" (UID: "20d12222-f3c4-41b1-8fca-53b6083a57e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:12:55.906807 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.906783    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "20d12222-f3c4-41b1-8fca-53b6083a57e3" (UID: "20d12222-f3c4-41b1-8fca-53b6083a57e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:12:55.907318 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.907293    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-workload-socket\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:12:55.907429 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.907321    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d12222-f3c4-41b1-8fca-53b6083a57e3-istiod-ca-cert\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:12:55.907429 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.907336    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-data\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:12:55.907429 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.907348    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-credential-socket\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:12:55.907429 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.907362    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-workload-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:12:55.908539 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.908509    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "20d12222-f3c4-41b1-8fca-53b6083a57e3" (UID: "20d12222-f3c4-41b1-8fca-53b6083a57e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 17:12:55.908641 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.908602    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "20d12222-f3c4-41b1-8fca-53b6083a57e3" (UID: "20d12222-f3c4-41b1-8fca-53b6083a57e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:12:55.908641 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.908612    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d12222-f3c4-41b1-8fca-53b6083a57e3-kube-api-access-mg66j" (OuterVolumeSpecName: "kube-api-access-mg66j") pod "20d12222-f3c4-41b1-8fca-53b6083a57e3" (UID: "20d12222-f3c4-41b1-8fca-53b6083a57e3"). InnerVolumeSpecName "kube-api-access-mg66j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:12:55.909020 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.909001    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "20d12222-f3c4-41b1-8fca-53b6083a57e3" (UID: "20d12222-f3c4-41b1-8fca-53b6083a57e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:12:55.927171 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.927146    2554 generic.go:356] "Generic (PLEG): container finished" podID="20d12222-f3c4-41b1-8fca-53b6083a57e3" containerID="21eec874f9162e60cf0a2c470ae43187b54c17065ca5c5de4654c5e884165093" exitCode=0
Jul 18 17:12:55.927277 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.927191    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"
Jul 18 17:12:55.927277 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.927210    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8" event={"ID":"20d12222-f3c4-41b1-8fca-53b6083a57e3","Type":"ContainerDied","Data":"21eec874f9162e60cf0a2c470ae43187b54c17065ca5c5de4654c5e884165093"}
Jul 18 17:12:55.927277 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.927252    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8" event={"ID":"20d12222-f3c4-41b1-8fca-53b6083a57e3","Type":"ContainerDied","Data":"0768f608f6327c90de65de024191b2de3d73a9c7d746b82662e4e740b4a37b67"}
Jul 18 17:12:55.927277 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.927274    2554 scope.go:117] "RemoveContainer" containerID="21eec874f9162e60cf0a2c470ae43187b54c17065ca5c5de4654c5e884165093"
Jul 18 17:12:55.935891 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.935872    2554 scope.go:117] "RemoveContainer" containerID="f5acb5dd8a20f863a62e16a9a058dd060aa993e2ce2dea2966ecfa4a1d2014c2"
Jul 18 17:12:55.946453 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.946431    2554 scope.go:117] "RemoveContainer" containerID="21eec874f9162e60cf0a2c470ae43187b54c17065ca5c5de4654c5e884165093"
Jul 18 17:12:55.946753 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:12:55.946721    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21eec874f9162e60cf0a2c470ae43187b54c17065ca5c5de4654c5e884165093\": container with ID starting with 21eec874f9162e60cf0a2c470ae43187b54c17065ca5c5de4654c5e884165093 not found: ID does not exist" containerID="21eec874f9162e60cf0a2c470ae43187b54c17065ca5c5de4654c5e884165093"
Jul 18 17:12:55.946866 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.946765    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21eec874f9162e60cf0a2c470ae43187b54c17065ca5c5de4654c5e884165093"} err="failed to get container status \"21eec874f9162e60cf0a2c470ae43187b54c17065ca5c5de4654c5e884165093\": rpc error: code = NotFound desc = could not find container \"21eec874f9162e60cf0a2c470ae43187b54c17065ca5c5de4654c5e884165093\": container with ID starting with 21eec874f9162e60cf0a2c470ae43187b54c17065ca5c5de4654c5e884165093 not found: ID does not exist"
Jul 18 17:12:55.946866 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.946791    2554 scope.go:117] "RemoveContainer" containerID="f5acb5dd8a20f863a62e16a9a058dd060aa993e2ce2dea2966ecfa4a1d2014c2"
Jul 18 17:12:55.947388 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:12:55.947363    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5acb5dd8a20f863a62e16a9a058dd060aa993e2ce2dea2966ecfa4a1d2014c2\": container with ID starting with f5acb5dd8a20f863a62e16a9a058dd060aa993e2ce2dea2966ecfa4a1d2014c2 not found: ID does not exist" containerID="f5acb5dd8a20f863a62e16a9a058dd060aa993e2ce2dea2966ecfa4a1d2014c2"
Jul 18 17:12:55.947537 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.947514    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5acb5dd8a20f863a62e16a9a058dd060aa993e2ce2dea2966ecfa4a1d2014c2"} err="failed to get container status \"f5acb5dd8a20f863a62e16a9a058dd060aa993e2ce2dea2966ecfa4a1d2014c2\": rpc error: code = NotFound desc = could not find container \"f5acb5dd8a20f863a62e16a9a058dd060aa993e2ce2dea2966ecfa4a1d2014c2\": container with ID starting with f5acb5dd8a20f863a62e16a9a058dd060aa993e2ce2dea2966ecfa4a1d2014c2 not found: ID does not exist"
Jul 18 17:12:55.948667 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.948649    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"]
Jul 18 17:12:55.952649 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:55.952630    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-47d19a6a-openshift-default-76c6c59755-ldtm8"]
Jul 18 17:12:56.008763 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:56.008731    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-envoy\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:12:56.008763 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:56.008757    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-podinfo\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:12:56.008763 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:56.008765    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d12222-f3c4-41b1-8fca-53b6083a57e3-istio-token\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:12:56.009014 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:56.008773    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg66j\" (UniqueName: \"kubernetes.io/projected/20d12222-f3c4-41b1-8fca-53b6083a57e3-kube-api-access-mg66j\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:12:56.780901 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:56.780864    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:56.846893 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:56.846863    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:56.929593 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:56.929551    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:57.028522 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:57.028491    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:57.185119 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:57.185063    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:12:57.766110 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:12:57.766083    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20d12222-f3c4-41b1-8fca-53b6083a57e3" path="/var/lib/kubelet/pods/20d12222-f3c4-41b1-8fca-53b6083a57e3/volumes"
Jul 18 17:13:02.881988 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:02.881949    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:13:02.947142 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:02.947103    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:13:09.712859 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.712823    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:13:09.975362 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.975282    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"]
Jul 18 17:13:09.975598 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.975586    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cedb3f78-1ded-4614-8f2e-c83fc76ddce8" containerName="revoke-keys"
Jul 18 17:13:09.975644 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.975599    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cedb3f78-1ded-4614-8f2e-c83fc76ddce8" containerName="revoke-keys"
Jul 18 17:13:09.975644 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.975614    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20d12222-f3c4-41b1-8fca-53b6083a57e3" containerName="istio-proxy"
Jul 18 17:13:09.975644 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.975619    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="20d12222-f3c4-41b1-8fca-53b6083a57e3" containerName="istio-proxy"
Jul 18 17:13:09.975644 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.975627    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20d12222-f3c4-41b1-8fca-53b6083a57e3" containerName="istio-proxy"
Jul 18 17:13:09.975644 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.975633    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="20d12222-f3c4-41b1-8fca-53b6083a57e3" containerName="istio-proxy"
Jul 18 17:13:09.975644 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.975641    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4" containerName="maas-api"
Jul 18 17:13:09.975644 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.975646    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4" containerName="maas-api"
Jul 18 17:13:09.975879 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.975706    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="20d12222-f3c4-41b1-8fca-53b6083a57e3" containerName="istio-proxy"
Jul 18 17:13:09.975879 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.975714    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="cedb3f78-1ded-4614-8f2e-c83fc76ddce8" containerName="revoke-keys"
Jul 18 17:13:09.975879 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.975721    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c079e1e6-e0e3-4a41-b4ac-6a32dffaaae4" containerName="maas-api"
Jul 18 17:13:09.980016 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.979997    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:09.982986 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.982965    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-259b7916-openshift-default-dockercfg-zmnh2\""
Jul 18 17:13:09.983253 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.983237    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 18 17:13:09.994650 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:09.994626    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"]
Jul 18 17:13:10.008381 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.008357    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:13:10.132468 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.132422    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmbqn\" (UniqueName: \"kubernetes.io/projected/b7936084-3ed3-4dd7-add8-d1782b9519c8-kube-api-access-qmbqn\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.132468 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.132474    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-workload-socket\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.132770 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.132502    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-envoy\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.132770 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.132540    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7936084-3ed3-4dd7-add8-d1782b9519c8-istiod-ca-cert\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.132770 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.132578    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-podinfo\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.132770 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.132609    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-token\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.132770 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.132645    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-credential-socket\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.132770 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.132670    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-data\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.132770 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.132686    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-workload-certs\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.233332 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.233244    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-envoy\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.233332 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.233284    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7936084-3ed3-4dd7-add8-d1782b9519c8-istiod-ca-cert\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.233332 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.233323    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-podinfo\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.233587 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.233355    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-token\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.233587 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.233500    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-credential-socket\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.233700 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.233585    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-data\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.233700 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.233620    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-workload-certs\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.233700 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.233656    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmbqn\" (UniqueName: \"kubernetes.io/projected/b7936084-3ed3-4dd7-add8-d1782b9519c8-kube-api-access-qmbqn\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.233700 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.233685    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-workload-socket\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.233964 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.233839    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-credential-socket\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.234021 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.233968    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-workload-certs\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.234021 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.234006    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-data\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.234119 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.234051    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-workload-socket\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.234170 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.234120    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7936084-3ed3-4dd7-add8-d1782b9519c8-istiod-ca-cert\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.236301 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.236274    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-envoy\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.236606 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.236586    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-podinfo\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.242208 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.242187    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-token\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.242373 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.242352    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmbqn\" (UniqueName: \"kubernetes.io/projected/b7936084-3ed3-4dd7-add8-d1782b9519c8-kube-api-access-qmbqn\") pod \"e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") " pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.252528 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.252503    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:13:10.289659 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.289629    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:10.466625 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.466556    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"]
Jul 18 17:13:10.469426 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:13:10.469402    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7936084_3ed3_4dd7_add8_d1782b9519c8.slice/crio-89bd9773a625f017b7e5fa2e1db325048f72137630f2ff066c60d3ec68494a65 WatchSource:0}: Error finding container 89bd9773a625f017b7e5fa2e1db325048f72137630f2ff066c60d3ec68494a65: Status 404 returned error can't find the container with id 89bd9773a625f017b7e5fa2e1db325048f72137630f2ff066c60d3ec68494a65
Jul 18 17:13:10.471477 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.471446    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:13:10.471542 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.471514    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:13:10.471578 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.471543    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:13:10.997025 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.996988    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9" event={"ID":"b7936084-3ed3-4dd7-add8-d1782b9519c8","Type":"ContainerStarted","Data":"c39c8423e2c4629929489b216d57aa28c2987fc3a3c9504799deb6ed10a94771"}
Jul 18 17:13:10.997437 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:10.997067    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9" event={"ID":"b7936084-3ed3-4dd7-add8-d1782b9519c8","Type":"ContainerStarted","Data":"89bd9773a625f017b7e5fa2e1db325048f72137630f2ff066c60d3ec68494a65"}
Jul 18 17:13:11.031935 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:11.031873    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9" podStartSLOduration=2.031822637 podStartE2EDuration="2.031822637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:13:11.030334331 +0000 UTC m=+2209.728165015" watchObservedRunningTime="2026-07-18 17:13:11.031822637 +0000 UTC m=+2209.729653323"
Jul 18 17:13:11.290207 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:11.290119    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:11.295146 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:11.295121    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:12.001603 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:12.001560    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:12.002648 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:12.002629    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:13.076485 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.076451    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:13:13.173381 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.173350    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:13:13.354284 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.354249    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"]
Jul 18 17:13:13.354635 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.354623    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="20d12222-f3c4-41b1-8fca-53b6083a57e3" containerName="istio-proxy"
Jul 18 17:13:13.357556 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.357539    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"
Jul 18 17:13:13.360338 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.360315    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 18 17:13:13.360571 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.360550    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-259b7916-serving-cert\""
Jul 18 17:13:13.361452 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.361433    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2pqsv\""
Jul 18 17:13:13.361521 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.361443    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 18 17:13:13.375667 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.375641    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"]
Jul 18 17:13:13.403496 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.403466    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:13:13.462596 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.462568    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5qjg\" (UniqueName: \"kubernetes.io/projected/dbfe5d18-c242-4553-9c44-b2800b86264b-kube-api-access-q5qjg\") pod \"maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff\" (UID: \"dbfe5d18-c242-4553-9c44-b2800b86264b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"
Jul 18 17:13:13.462709 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.462617    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbfe5d18-c242-4553-9c44-b2800b86264b-maas-api-tls\") pod \"maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff\" (UID: \"dbfe5d18-c242-4553-9c44-b2800b86264b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"
Jul 18 17:13:13.462709 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.462691    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe5d18-c242-4553-9c44-b2800b86264b-openshift-service-ca\") pod \"maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff\" (UID: \"dbfe5d18-c242-4553-9c44-b2800b86264b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"
Jul 18 17:13:13.505635 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.505610    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:13:13.563369 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.563333    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbfe5d18-c242-4553-9c44-b2800b86264b-maas-api-tls\") pod \"maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff\" (UID: \"dbfe5d18-c242-4553-9c44-b2800b86264b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"
Jul 18 17:13:13.563531 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.563389    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe5d18-c242-4553-9c44-b2800b86264b-openshift-service-ca\") pod \"maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff\" (UID: \"dbfe5d18-c242-4553-9c44-b2800b86264b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"
Jul 18 17:13:13.563531 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.563447    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5qjg\" (UniqueName: \"kubernetes.io/projected/dbfe5d18-c242-4553-9c44-b2800b86264b-kube-api-access-q5qjg\") pod \"maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff\" (UID: \"dbfe5d18-c242-4553-9c44-b2800b86264b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"
Jul 18 17:13:13.564091 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.564067    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe5d18-c242-4553-9c44-b2800b86264b-openshift-service-ca\") pod \"maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff\" (UID: \"dbfe5d18-c242-4553-9c44-b2800b86264b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"
Jul 18 17:13:13.565971 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.565950    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbfe5d18-c242-4553-9c44-b2800b86264b-maas-api-tls\") pod \"maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff\" (UID: \"dbfe5d18-c242-4553-9c44-b2800b86264b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"
Jul 18 17:13:13.577063 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.577042    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5qjg\" (UniqueName: \"kubernetes.io/projected/dbfe5d18-c242-4553-9c44-b2800b86264b-kube-api-access-q5qjg\") pod \"maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff\" (UID: \"dbfe5d18-c242-4553-9c44-b2800b86264b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"
Jul 18 17:13:13.668270 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.668187    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"
Jul 18 17:13:13.829124 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.829015    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"]
Jul 18 17:13:13.832041 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:13:13.832019    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfe5d18_c242_4553_9c44_b2800b86264b.slice/crio-03be11854f41c836089db5c04fdc73d1c3949fc66487171b82cb0bff335c58d6 WatchSource:0}: Error finding container 03be11854f41c836089db5c04fdc73d1c3949fc66487171b82cb0bff335c58d6: Status 404 returned error can't find the container with id 03be11854f41c836089db5c04fdc73d1c3949fc66487171b82cb0bff335c58d6
Jul 18 17:13:13.834027 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.834007    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 17:13:13.995629 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:13.995542    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:13:14.012451 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:14.012420    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff" event={"ID":"dbfe5d18-c242-4553-9c44-b2800b86264b","Type":"ContainerStarted","Data":"03be11854f41c836089db5c04fdc73d1c3949fc66487171b82cb0bff335c58d6"}
Jul 18 17:13:14.105198 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:14.105158    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:13:15.018475 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:15.018442    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff" event={"ID":"dbfe5d18-c242-4553-9c44-b2800b86264b","Type":"ContainerStarted","Data":"4af0babffa25b33679703ff18a217becadecb791c765b8d932c99cefbf5d4149"}
Jul 18 17:13:15.018680 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:15.018574    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"
Jul 18 17:13:15.038977 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:15.038932    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff" podStartSLOduration=1.5716798330000001 podStartE2EDuration="2.038916831s" totalImagesPullingTime="467.236998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:13:15.038260854 +0000 UTC m=+2213.736091576" watchObservedRunningTime="2026-07-18 17:13:15.038916831 +0000 UTC m=+2213.736747515"
Jul 18 17:13:17.466036 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:17.465998    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:13:21.029700 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:21.029661    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"
Jul 18 17:13:23.467088 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.467043    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"]
Jul 18 17:13:23.472072 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.472052    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.475094 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.475064    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-259b7916\"/\"openshift-service-ca.crt\""
Jul 18 17:13:23.475094 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.475077    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-259b7916\"/\"kube-root-ca.crt\""
Jul 18 17:13:23.475899 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.475874    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-259b7916\"/\"default-dockercfg-rw456\""
Jul 18 17:13:23.476035 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.475899    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-259b7916\"/\"e2e-lifecycle-model-259b7916-kserve-self-signed-certs\""
Jul 18 17:13:23.480768 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.480745    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"]
Jul 18 17:13:23.552111 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.552071    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-home\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.552295 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.552133    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw5mm\" (UniqueName: \"kubernetes.io/projected/77887392-1fcf-43ff-bd47-0f822f4a2340-kube-api-access-dw5mm\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.552295 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.552223    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77887392-1fcf-43ff-bd47-0f822f4a2340-tls-certs\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.552295 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.552292    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-tmp-dir\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.552467 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.552318    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-dshm\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.552467 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.552347    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-model-cache\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.558245 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.558216    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:13:23.653644 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.653556    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-tmp-dir\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.653644 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.653599    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-dshm\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.653644 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.653621    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-model-cache\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.653959 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.653721    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-home\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.653959 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.653773    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw5mm\" (UniqueName: \"kubernetes.io/projected/77887392-1fcf-43ff-bd47-0f822f4a2340-kube-api-access-dw5mm\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.653959 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.653823    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77887392-1fcf-43ff-bd47-0f822f4a2340-tls-certs\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.654109 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.654009    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-tmp-dir\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.654109 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.654063    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-model-cache\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.654189 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.654174    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-home\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.656316 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.656295    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77887392-1fcf-43ff-bd47-0f822f4a2340-tls-certs\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.656428 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.656412    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-dshm\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.662225 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.662203    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw5mm\" (UniqueName: \"kubernetes.io/projected/77887392-1fcf-43ff-bd47-0f822f4a2340-kube-api-access-dw5mm\") pod \"e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") " pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.783453 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.783419    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:23.923545 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:23.923519    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"]
Jul 18 17:13:23.925930 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:13:23.925900    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77887392_1fcf_43ff_bd47_0f822f4a2340.slice/crio-2cc6e5b54ee436a436ef12e8921ba242883ef273a2ef63030611145bd99692b8 WatchSource:0}: Error finding container 2cc6e5b54ee436a436ef12e8921ba242883ef273a2ef63030611145bd99692b8: Status 404 returned error can't find the container with id 2cc6e5b54ee436a436ef12e8921ba242883ef273a2ef63030611145bd99692b8
Jul 18 17:13:24.061195 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:24.061161    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c" event={"ID":"77887392-1fcf-43ff-bd47-0f822f4a2340","Type":"ContainerStarted","Data":"5e11ff71cf8c55eeaef589dace97378a48b6979ac4affc306aef205a8cb84eaa"}
Jul 18 17:13:24.061357 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:24.061209    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c" event={"ID":"77887392-1fcf-43ff-bd47-0f822f4a2340","Type":"ContainerStarted","Data":"2cc6e5b54ee436a436ef12e8921ba242883ef273a2ef63030611145bd99692b8"}
Jul 18 17:13:24.083300 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:24.083246    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c" podStartSLOduration=1.083230494 podStartE2EDuration="1.083230494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:13:24.081588754 +0000 UTC m=+2222.779419438" watchObservedRunningTime="2026-07-18 17:13:24.083230494 +0000 UTC m=+2222.781061178"
Jul 18 17:13:26.073091 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:26.073057    2554 generic.go:356] "Generic (PLEG): container finished" podID="b7936084-3ed3-4dd7-add8-d1782b9519c8" containerID="c39c8423e2c4629929489b216d57aa28c2987fc3a3c9504799deb6ed10a94771" exitCode=0
Jul 18 17:13:26.073520 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:26.073131    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9" event={"ID":"b7936084-3ed3-4dd7-add8-d1782b9519c8","Type":"ContainerDied","Data":"c39c8423e2c4629929489b216d57aa28c2987fc3a3c9504799deb6ed10a94771"}
Jul 18 17:13:26.073520 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:26.073505    2554 scope.go:117] "RemoveContainer" containerID="c39c8423e2c4629929489b216d57aa28c2987fc3a3c9504799deb6ed10a94771"
Jul 18 17:13:26.074461 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:26.074433    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:13:26.074537 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:26.074524    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:13:26.074577 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:26.074561    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:13:27.003998 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:27.003961    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:27.083830 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:27.083783    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9" event={"ID":"b7936084-3ed3-4dd7-add8-d1782b9519c8","Type":"ContainerStarted","Data":"f2a8e540da65ca8dd408c329b3ac05755e0618dc730dcf3911746efe5d21e00c"}
Jul 18 17:13:27.290058 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:27.289964    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:27.291250 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:27.291220    2554 patch_prober.go:28] interesting pod/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 18 17:13:27.291390 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:27.291269    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9" podUID="b7936084-3ed3-4dd7-add8-d1782b9519c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 18 17:13:28.290067 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:28.290023    2554 patch_prober.go:28] interesting pod/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 18 17:13:28.290531 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:28.290097    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9" podUID="b7936084-3ed3-4dd7-add8-d1782b9519c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 18 17:13:29.290350 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:29.290302    2554 patch_prober.go:28] interesting pod/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 18 17:13:29.290886 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:29.290367    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9" podUID="b7936084-3ed3-4dd7-add8-d1782b9519c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 18 17:13:30.290877 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:30.290820    2554 patch_prober.go:28] interesting pod/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 18 17:13:30.291262 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:30.290904    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9" podUID="b7936084-3ed3-4dd7-add8-d1782b9519c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 18 17:13:31.294321 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:31.294288    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:31.294702 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:31.294574    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:31.295223 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:31.295203    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:13:33.783744 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:33.783710    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:33.784194 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:33.783760    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:33.796311 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:33.796285    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:34.134827 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:34.134795    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:13:56.725714 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:56.725634    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"]
Jul 18 17:13:56.726222 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:56.725891    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff" podUID="dbfe5d18-c242-4553-9c44-b2800b86264b" containerName="maas-api" containerID="cri-o://4af0babffa25b33679703ff18a217becadecb791c765b8d932c99cefbf5d4149" gracePeriod=30
Jul 18 17:13:56.753183 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:56.753158    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:13:56.966626 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:56.966600    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"
Jul 18 17:13:57.031327 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.031254    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5qjg\" (UniqueName: \"kubernetes.io/projected/dbfe5d18-c242-4553-9c44-b2800b86264b-kube-api-access-q5qjg\") pod \"dbfe5d18-c242-4553-9c44-b2800b86264b\" (UID: \"dbfe5d18-c242-4553-9c44-b2800b86264b\") "
Jul 18 17:13:57.031327 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.031290    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe5d18-c242-4553-9c44-b2800b86264b-openshift-service-ca\") pod \"dbfe5d18-c242-4553-9c44-b2800b86264b\" (UID: \"dbfe5d18-c242-4553-9c44-b2800b86264b\") "
Jul 18 17:13:57.031327 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.031317    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbfe5d18-c242-4553-9c44-b2800b86264b-maas-api-tls\") pod \"dbfe5d18-c242-4553-9c44-b2800b86264b\" (UID: \"dbfe5d18-c242-4553-9c44-b2800b86264b\") "
Jul 18 17:13:57.031663 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.031641    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfe5d18-c242-4553-9c44-b2800b86264b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dbfe5d18-c242-4553-9c44-b2800b86264b" (UID: "dbfe5d18-c242-4553-9c44-b2800b86264b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:13:57.033450 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.033424    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfe5d18-c242-4553-9c44-b2800b86264b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dbfe5d18-c242-4553-9c44-b2800b86264b" (UID: "dbfe5d18-c242-4553-9c44-b2800b86264b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:13:57.033543 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.033480    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfe5d18-c242-4553-9c44-b2800b86264b-kube-api-access-q5qjg" (OuterVolumeSpecName: "kube-api-access-q5qjg") pod "dbfe5d18-c242-4553-9c44-b2800b86264b" (UID: "dbfe5d18-c242-4553-9c44-b2800b86264b"). InnerVolumeSpecName "kube-api-access-q5qjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:13:57.132312 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.132276    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbfe5d18-c242-4553-9c44-b2800b86264b-maas-api-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:13:57.132312 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.132309    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5qjg\" (UniqueName: \"kubernetes.io/projected/dbfe5d18-c242-4553-9c44-b2800b86264b-kube-api-access-q5qjg\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:13:57.132312 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.132319    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbfe5d18-c242-4553-9c44-b2800b86264b-openshift-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:13:57.230229 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.230191    2554 generic.go:356] "Generic (PLEG): container finished" podID="dbfe5d18-c242-4553-9c44-b2800b86264b" containerID="4af0babffa25b33679703ff18a217becadecb791c765b8d932c99cefbf5d4149" exitCode=0
Jul 18 17:13:57.230391 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.230240    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"
Jul 18 17:13:57.230391 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.230274    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff" event={"ID":"dbfe5d18-c242-4553-9c44-b2800b86264b","Type":"ContainerDied","Data":"4af0babffa25b33679703ff18a217becadecb791c765b8d932c99cefbf5d4149"}
Jul 18 17:13:57.230391 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.230331    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff" event={"ID":"dbfe5d18-c242-4553-9c44-b2800b86264b","Type":"ContainerDied","Data":"03be11854f41c836089db5c04fdc73d1c3949fc66487171b82cb0bff335c58d6"}
Jul 18 17:13:57.230391 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.230356    2554 scope.go:117] "RemoveContainer" containerID="4af0babffa25b33679703ff18a217becadecb791c765b8d932c99cefbf5d4149"
Jul 18 17:13:57.239431 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.239415    2554 scope.go:117] "RemoveContainer" containerID="4af0babffa25b33679703ff18a217becadecb791c765b8d932c99cefbf5d4149"
Jul 18 17:13:57.239672 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:13:57.239652    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af0babffa25b33679703ff18a217becadecb791c765b8d932c99cefbf5d4149\": container with ID starting with 4af0babffa25b33679703ff18a217becadecb791c765b8d932c99cefbf5d4149 not found: ID does not exist" containerID="4af0babffa25b33679703ff18a217becadecb791c765b8d932c99cefbf5d4149"
Jul 18 17:13:57.239718 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.239679    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af0babffa25b33679703ff18a217becadecb791c765b8d932c99cefbf5d4149"} err="failed to get container status \"4af0babffa25b33679703ff18a217becadecb791c765b8d932c99cefbf5d4149\": rpc error: code = NotFound desc = could not find container \"4af0babffa25b33679703ff18a217becadecb791c765b8d932c99cefbf5d4149\": container with ID starting with 4af0babffa25b33679703ff18a217becadecb791c765b8d932c99cefbf5d4149 not found: ID does not exist"
Jul 18 17:13:57.251511 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.251487    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"]
Jul 18 17:13:57.253554 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.253535    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-259b7916-65c54fb8f6-fhvff"]
Jul 18 17:13:57.765796 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:13:57.765764    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbfe5d18-c242-4553-9c44-b2800b86264b" path="/var/lib/kubelet/pods/dbfe5d18-c242-4553-9c44-b2800b86264b/volumes"
Jul 18 17:14:01.302409 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:01.302374    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:03.657053 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:03.657014    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:08.598639 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:08.598590    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"]
Jul 18 17:14:08.673748 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:08.673713    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:21.943398 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:21.943365    2554 scope.go:117] "RemoveContainer" containerID="4ecb5a189829873dd1cd04c152f990641a05f0d6fe9d5cd4cc289576ad7029c7"
Jul 18 17:14:23.619912 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.619812    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c" podUID="77887392-1fcf-43ff-bd47-0f822f4a2340" containerName="main" containerID="cri-o://5e11ff71cf8c55eeaef589dace97378a48b6979ac4affc306aef205a8cb84eaa" gracePeriod=45
Jul 18 17:14:23.861592 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.861568    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:14:23.976184 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.976092    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-home\") pod \"77887392-1fcf-43ff-bd47-0f822f4a2340\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") "
Jul 18 17:14:23.976184 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.976126    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-dshm\") pod \"77887392-1fcf-43ff-bd47-0f822f4a2340\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") "
Jul 18 17:14:23.976184 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.976171    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77887392-1fcf-43ff-bd47-0f822f4a2340-tls-certs\") pod \"77887392-1fcf-43ff-bd47-0f822f4a2340\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") "
Jul 18 17:14:23.976459 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.976196    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-model-cache\") pod \"77887392-1fcf-43ff-bd47-0f822f4a2340\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") "
Jul 18 17:14:23.976459 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.976283    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-tmp-dir\") pod \"77887392-1fcf-43ff-bd47-0f822f4a2340\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") "
Jul 18 17:14:23.976459 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.976326    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw5mm\" (UniqueName: \"kubernetes.io/projected/77887392-1fcf-43ff-bd47-0f822f4a2340-kube-api-access-dw5mm\") pod \"77887392-1fcf-43ff-bd47-0f822f4a2340\" (UID: \"77887392-1fcf-43ff-bd47-0f822f4a2340\") "
Jul 18 17:14:23.976614 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.976447    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-home" (OuterVolumeSpecName: "home") pod "77887392-1fcf-43ff-bd47-0f822f4a2340" (UID: "77887392-1fcf-43ff-bd47-0f822f4a2340"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:14:23.976614 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.976542    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77887392-1fcf-43ff-bd47-0f822f4a2340" (UID: "77887392-1fcf-43ff-bd47-0f822f4a2340"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:14:23.976614 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.976553    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-model-cache" (OuterVolumeSpecName: "model-cache") pod "77887392-1fcf-43ff-bd47-0f822f4a2340" (UID: "77887392-1fcf-43ff-bd47-0f822f4a2340"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:14:23.976731 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.976652    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-home\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:14:23.976731 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.976671    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-model-cache\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:14:23.976731 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.976686    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-tmp-dir\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:14:23.978662 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.978635    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77887392-1fcf-43ff-bd47-0f822f4a2340-kube-api-access-dw5mm" (OuterVolumeSpecName: "kube-api-access-dw5mm") pod "77887392-1fcf-43ff-bd47-0f822f4a2340" (UID: "77887392-1fcf-43ff-bd47-0f822f4a2340"). InnerVolumeSpecName "kube-api-access-dw5mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:14:23.978880 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.978832    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77887392-1fcf-43ff-bd47-0f822f4a2340-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77887392-1fcf-43ff-bd47-0f822f4a2340" (UID: "77887392-1fcf-43ff-bd47-0f822f4a2340"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:14:23.978880 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:23.978832    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-dshm" (OuterVolumeSpecName: "dshm") pod "77887392-1fcf-43ff-bd47-0f822f4a2340" (UID: "77887392-1fcf-43ff-bd47-0f822f4a2340"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:14:24.077397 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:24.077349    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77887392-1fcf-43ff-bd47-0f822f4a2340-tls-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:14:24.077397 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:24.077390    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw5mm\" (UniqueName: \"kubernetes.io/projected/77887392-1fcf-43ff-bd47-0f822f4a2340-kube-api-access-dw5mm\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:14:24.077397 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:24.077400    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77887392-1fcf-43ff-bd47-0f822f4a2340-dshm\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:14:24.359782 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:24.359744    2554 generic.go:356] "Generic (PLEG): container finished" podID="77887392-1fcf-43ff-bd47-0f822f4a2340" containerID="5e11ff71cf8c55eeaef589dace97378a48b6979ac4affc306aef205a8cb84eaa" exitCode=0
Jul 18 17:14:24.360008 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:24.359793    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"
Jul 18 17:14:24.360008 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:24.359806    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c" event={"ID":"77887392-1fcf-43ff-bd47-0f822f4a2340","Type":"ContainerDied","Data":"5e11ff71cf8c55eeaef589dace97378a48b6979ac4affc306aef205a8cb84eaa"}
Jul 18 17:14:24.360008 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:24.359876    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c" event={"ID":"77887392-1fcf-43ff-bd47-0f822f4a2340","Type":"ContainerDied","Data":"2cc6e5b54ee436a436ef12e8921ba242883ef273a2ef63030611145bd99692b8"}
Jul 18 17:14:24.360008 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:24.359901    2554 scope.go:117] "RemoveContainer" containerID="5e11ff71cf8c55eeaef589dace97378a48b6979ac4affc306aef205a8cb84eaa"
Jul 18 17:14:24.369211 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:24.369106    2554 scope.go:117] "RemoveContainer" containerID="5e11ff71cf8c55eeaef589dace97378a48b6979ac4affc306aef205a8cb84eaa"
Jul 18 17:14:24.369402 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:14:24.369383    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e11ff71cf8c55eeaef589dace97378a48b6979ac4affc306aef205a8cb84eaa\": container with ID starting with 5e11ff71cf8c55eeaef589dace97378a48b6979ac4affc306aef205a8cb84eaa not found: ID does not exist" containerID="5e11ff71cf8c55eeaef589dace97378a48b6979ac4affc306aef205a8cb84eaa"
Jul 18 17:14:24.369446 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:24.369413    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e11ff71cf8c55eeaef589dace97378a48b6979ac4affc306aef205a8cb84eaa"} err="failed to get container status \"5e11ff71cf8c55eeaef589dace97378a48b6979ac4affc306aef205a8cb84eaa\": rpc error: code = NotFound desc = could not find container \"5e11ff71cf8c55eeaef589dace97378a48b6979ac4affc306aef205a8cb84eaa\": container with ID starting with 5e11ff71cf8c55eeaef589dace97378a48b6979ac4affc306aef205a8cb84eaa not found: ID does not exist"
Jul 18 17:14:24.388134 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:24.388111    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"]
Jul 18 17:14:24.396348 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:24.396320    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-259b7916/e2e-lifecycle-model-259b7916-kserve-67fd4665c5-xpr5c"]
Jul 18 17:14:25.765804 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:25.765770    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77887392-1fcf-43ff-bd47-0f822f4a2340" path="/var/lib/kubelet/pods/77887392-1fcf-43ff-bd47-0f822f4a2340/volumes"
Jul 18 17:14:45.067992 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:45.067956    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:45.931011 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:45.930982    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:51.534294 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:51.534253    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:51.614277 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:51.614242    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:51.805449 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:51.805360    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"]
Jul 18 17:14:51.805596 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:51.805579    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9" podUID="b7936084-3ed3-4dd7-add8-d1782b9519c8" containerName="istio-proxy" containerID="cri-o://f2a8e540da65ca8dd408c329b3ac05755e0618dc730dcf3911746efe5d21e00c" gracePeriod=30
Jul 18 17:14:51.820176 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:51.820146    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:52.155676 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:52.155642    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:53.162401 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:53.162355    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:53.262044 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:53.262012    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:53.366762 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:53.366719    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:54.151753 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:54.151720    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:54.251938 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:54.251868    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:54.353081 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:54.353026    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:55.687252 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:55.687220    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:14:57.068566 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.068494    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:14:57.162957 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.162919    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-credential-socket\") pod \"b7936084-3ed3-4dd7-add8-d1782b9519c8\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") "
Jul 18 17:14:57.162957 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.162961    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-workload-socket\") pod \"b7936084-3ed3-4dd7-add8-d1782b9519c8\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") "
Jul 18 17:14:57.163212 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.162978    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-workload-certs\") pod \"b7936084-3ed3-4dd7-add8-d1782b9519c8\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") "
Jul 18 17:14:57.163212 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.163005    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-data\") pod \"b7936084-3ed3-4dd7-add8-d1782b9519c8\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") "
Jul 18 17:14:57.163212 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.163033    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-envoy\") pod \"b7936084-3ed3-4dd7-add8-d1782b9519c8\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") "
Jul 18 17:14:57.163212 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.163070    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7936084-3ed3-4dd7-add8-d1782b9519c8-istiod-ca-cert\") pod \"b7936084-3ed3-4dd7-add8-d1782b9519c8\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") "
Jul 18 17:14:57.163212 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.163100    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-token\") pod \"b7936084-3ed3-4dd7-add8-d1782b9519c8\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") "
Jul 18 17:14:57.163212 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.163161    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmbqn\" (UniqueName: \"kubernetes.io/projected/b7936084-3ed3-4dd7-add8-d1782b9519c8-kube-api-access-qmbqn\") pod \"b7936084-3ed3-4dd7-add8-d1782b9519c8\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") "
Jul 18 17:14:57.163212 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.163193    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-podinfo\") pod \"b7936084-3ed3-4dd7-add8-d1782b9519c8\" (UID: \"b7936084-3ed3-4dd7-add8-d1782b9519c8\") "
Jul 18 17:14:57.163547 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.163264    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b7936084-3ed3-4dd7-add8-d1782b9519c8" (UID: "b7936084-3ed3-4dd7-add8-d1782b9519c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:14:57.163547 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.163299    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b7936084-3ed3-4dd7-add8-d1782b9519c8" (UID: "b7936084-3ed3-4dd7-add8-d1782b9519c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:14:57.163547 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.163307    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b7936084-3ed3-4dd7-add8-d1782b9519c8" (UID: "b7936084-3ed3-4dd7-add8-d1782b9519c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:14:57.163547 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.163521    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-credential-socket\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:14:57.163547 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.163541    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-workload-socket\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:14:57.163787 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.163555    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-workload-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:14:57.163787 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.163554    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7936084-3ed3-4dd7-add8-d1782b9519c8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b7936084-3ed3-4dd7-add8-d1782b9519c8" (UID: "b7936084-3ed3-4dd7-add8-d1782b9519c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:14:57.164081 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.163960    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-data" (OuterVolumeSpecName: "istio-data") pod "b7936084-3ed3-4dd7-add8-d1782b9519c8" (UID: "b7936084-3ed3-4dd7-add8-d1782b9519c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:14:57.165608 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.165580    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b7936084-3ed3-4dd7-add8-d1782b9519c8" (UID: "b7936084-3ed3-4dd7-add8-d1782b9519c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 17:14:57.165608 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.165586    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "b7936084-3ed3-4dd7-add8-d1782b9519c8" (UID: "b7936084-3ed3-4dd7-add8-d1782b9519c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:14:57.165760 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.165658    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7936084-3ed3-4dd7-add8-d1782b9519c8-kube-api-access-qmbqn" (OuterVolumeSpecName: "kube-api-access-qmbqn") pod "b7936084-3ed3-4dd7-add8-d1782b9519c8" (UID: "b7936084-3ed3-4dd7-add8-d1782b9519c8"). InnerVolumeSpecName "kube-api-access-qmbqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:14:57.165892 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.165871    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b7936084-3ed3-4dd7-add8-d1782b9519c8" (UID: "b7936084-3ed3-4dd7-add8-d1782b9519c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:14:57.264470 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.264444    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7936084-3ed3-4dd7-add8-d1782b9519c8-istiod-ca-cert\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:14:57.264470 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.264468    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-token\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:14:57.264470 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.264477    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmbqn\" (UniqueName: \"kubernetes.io/projected/b7936084-3ed3-4dd7-add8-d1782b9519c8-kube-api-access-qmbqn\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:14:57.264693 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.264486    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-podinfo\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:14:57.264693 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.264494    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-data\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:14:57.264693 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.264502    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7936084-3ed3-4dd7-add8-d1782b9519c8-istio-envoy\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:14:57.514386 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.514352    2554 generic.go:356] "Generic (PLEG): container finished" podID="b7936084-3ed3-4dd7-add8-d1782b9519c8" containerID="f2a8e540da65ca8dd408c329b3ac05755e0618dc730dcf3911746efe5d21e00c" exitCode=0
Jul 18 17:14:57.514577 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.514394    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"
Jul 18 17:14:57.514577 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.514441    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9" event={"ID":"b7936084-3ed3-4dd7-add8-d1782b9519c8","Type":"ContainerDied","Data":"f2a8e540da65ca8dd408c329b3ac05755e0618dc730dcf3911746efe5d21e00c"}
Jul 18 17:14:57.514577 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.514498    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9" event={"ID":"b7936084-3ed3-4dd7-add8-d1782b9519c8","Type":"ContainerDied","Data":"89bd9773a625f017b7e5fa2e1db325048f72137630f2ff066c60d3ec68494a65"}
Jul 18 17:14:57.514577 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.514527    2554 scope.go:117] "RemoveContainer" containerID="f2a8e540da65ca8dd408c329b3ac05755e0618dc730dcf3911746efe5d21e00c"
Jul 18 17:14:57.526027 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.526009    2554 scope.go:117] "RemoveContainer" containerID="c39c8423e2c4629929489b216d57aa28c2987fc3a3c9504799deb6ed10a94771"
Jul 18 17:14:57.533904 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.533881    2554 scope.go:117] "RemoveContainer" containerID="f2a8e540da65ca8dd408c329b3ac05755e0618dc730dcf3911746efe5d21e00c"
Jul 18 17:14:57.534190 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:14:57.534168    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a8e540da65ca8dd408c329b3ac05755e0618dc730dcf3911746efe5d21e00c\": container with ID starting with f2a8e540da65ca8dd408c329b3ac05755e0618dc730dcf3911746efe5d21e00c not found: ID does not exist" containerID="f2a8e540da65ca8dd408c329b3ac05755e0618dc730dcf3911746efe5d21e00c"
Jul 18 17:14:57.534286 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.534198    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a8e540da65ca8dd408c329b3ac05755e0618dc730dcf3911746efe5d21e00c"} err="failed to get container status \"f2a8e540da65ca8dd408c329b3ac05755e0618dc730dcf3911746efe5d21e00c\": rpc error: code = NotFound desc = could not find container \"f2a8e540da65ca8dd408c329b3ac05755e0618dc730dcf3911746efe5d21e00c\": container with ID starting with f2a8e540da65ca8dd408c329b3ac05755e0618dc730dcf3911746efe5d21e00c not found: ID does not exist"
Jul 18 17:14:57.534286 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.534215    2554 scope.go:117] "RemoveContainer" containerID="c39c8423e2c4629929489b216d57aa28c2987fc3a3c9504799deb6ed10a94771"
Jul 18 17:14:57.534473 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:14:57.534447    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39c8423e2c4629929489b216d57aa28c2987fc3a3c9504799deb6ed10a94771\": container with ID starting with c39c8423e2c4629929489b216d57aa28c2987fc3a3c9504799deb6ed10a94771 not found: ID does not exist" containerID="c39c8423e2c4629929489b216d57aa28c2987fc3a3c9504799deb6ed10a94771"
Jul 18 17:14:57.534513 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.534486    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39c8423e2c4629929489b216d57aa28c2987fc3a3c9504799deb6ed10a94771"} err="failed to get container status \"c39c8423e2c4629929489b216d57aa28c2987fc3a3c9504799deb6ed10a94771\": rpc error: code = NotFound desc = could not find container \"c39c8423e2c4629929489b216d57aa28c2987fc3a3c9504799deb6ed10a94771\": container with ID starting with c39c8423e2c4629929489b216d57aa28c2987fc3a3c9504799deb6ed10a94771 not found: ID does not exist"
Jul 18 17:14:57.547886 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.547840    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"]
Jul 18 17:14:57.565448 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.565412    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-259b7916-openshift-default-59db5577ff-v8fx9"]
Jul 18 17:14:57.765889 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:14:57.765796    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7936084-3ed3-4dd7-add8-d1782b9519c8" path="/var/lib/kubelet/pods/b7936084-3ed3-4dd7-add8-d1782b9519c8/volumes"
Jul 18 17:15:00.139170 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.139133    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739915-ffhzg"]
Jul 18 17:15:00.139553 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.139449    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7936084-3ed3-4dd7-add8-d1782b9519c8" containerName="istio-proxy"
Jul 18 17:15:00.139553 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.139460    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7936084-3ed3-4dd7-add8-d1782b9519c8" containerName="istio-proxy"
Jul 18 17:15:00.139553 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.139474    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbfe5d18-c242-4553-9c44-b2800b86264b" containerName="maas-api"
Jul 18 17:15:00.139553 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.139480    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbfe5d18-c242-4553-9c44-b2800b86264b" containerName="maas-api"
Jul 18 17:15:00.139553 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.139491    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77887392-1fcf-43ff-bd47-0f822f4a2340" containerName="main"
Jul 18 17:15:00.139553 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.139497    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="77887392-1fcf-43ff-bd47-0f822f4a2340" containerName="main"
Jul 18 17:15:00.139741 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.139570    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="77887392-1fcf-43ff-bd47-0f822f4a2340" containerName="main"
Jul 18 17:15:00.139741 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.139579    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbfe5d18-c242-4553-9c44-b2800b86264b" containerName="maas-api"
Jul 18 17:15:00.139741 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.139586    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7936084-3ed3-4dd7-add8-d1782b9519c8" containerName="istio-proxy"
Jul 18 17:15:00.139741 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.139591    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7936084-3ed3-4dd7-add8-d1782b9519c8" containerName="istio-proxy"
Jul 18 17:15:00.144196 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.144177    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739915-ffhzg"
Jul 18 17:15:00.147115 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.147093    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7rq8\""
Jul 18 17:15:00.149244 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.149220    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739915-ffhzg"]
Jul 18 17:15:00.155141 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.155123    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739915-ffhzg"
Jul 18 17:15:00.292785 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.292763    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29739915-ffhzg"]
Jul 18 17:15:00.295191 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:15:00.295166    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5a6238_5faa_4620_8182_98e628c6d47c.slice/crio-2e087a671d7c138a68cfb4d27f5d51a76d7ac300e46be0b6da87caf8accb1865 WatchSource:0}: Error finding container 2e087a671d7c138a68cfb4d27f5d51a76d7ac300e46be0b6da87caf8accb1865: Status 404 returned error can't find the container with id 2e087a671d7c138a68cfb4d27f5d51a76d7ac300e46be0b6da87caf8accb1865
Jul 18 17:15:00.530711 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:00.530678    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739915-ffhzg" event={"ID":"ce5a6238-5faa-4620-8182-98e628c6d47c","Type":"ContainerStarted","Data":"2e087a671d7c138a68cfb4d27f5d51a76d7ac300e46be0b6da87caf8accb1865"}
Jul 18 17:15:01.371255 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:01.371222    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:01.442149 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:01.442063    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:01.537035 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:01.537005    2554 generic.go:356] "Generic (PLEG): container finished" podID="ce5a6238-5faa-4620-8182-98e628c6d47c" containerID="13e2ff8e0391aefc439f087fe0c9b85911f857842d37d471dd3855a2cb9472fc" exitCode=0
Jul 18 17:15:01.537238 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:01.537099    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739915-ffhzg" event={"ID":"ce5a6238-5faa-4620-8182-98e628c6d47c","Type":"ContainerDied","Data":"13e2ff8e0391aefc439f087fe0c9b85911f857842d37d471dd3855a2cb9472fc"}
Jul 18 17:15:02.035862 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:02.035800    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:02.671516 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:02.671486    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739915-ffhzg"
Jul 18 17:15:03.546999 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:03.546969    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739915-ffhzg"
Jul 18 17:15:03.547168 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:03.546962    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29739915-ffhzg" event={"ID":"ce5a6238-5faa-4620-8182-98e628c6d47c","Type":"ContainerDied","Data":"2e087a671d7c138a68cfb4d27f5d51a76d7ac300e46be0b6da87caf8accb1865"}
Jul 18 17:15:03.547168 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:03.547110    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e087a671d7c138a68cfb4d27f5d51a76d7ac300e46be0b6da87caf8accb1865"
Jul 18 17:15:07.730050 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:07.730009    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:07.833132 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:07.833096    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:07.930224 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:07.930193    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:09.887169 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:09.887136    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:09.986485 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:09.986455    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:14.979622 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:14.979533    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:15.080525 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:15.080491    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:18.419725 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:18.419683    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:18.761325 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:18.761246    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:21.965822 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:21.965790    2554 scope.go:117] "RemoveContainer" containerID="04d23bed89052ee22515459e59341e9e71a6e77b922043c5bf7b022441d94996"
Jul 18 17:15:39.697218 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:39.697182    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:39.759924 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:39.759888    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:45.846775 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:45.846733    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:45.985287 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:45.985248    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:46.169600 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.169517    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:46.438662 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.438584    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"]
Jul 18 17:15:46.438989 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.438974    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce5a6238-5faa-4620-8182-98e628c6d47c" containerName="cleanup"
Jul 18 17:15:46.439047 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.438991    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce5a6238-5faa-4620-8182-98e628c6d47c" containerName="cleanup"
Jul 18 17:15:46.439047 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.439010    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7936084-3ed3-4dd7-add8-d1782b9519c8" containerName="istio-proxy"
Jul 18 17:15:46.439047 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.439016    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7936084-3ed3-4dd7-add8-d1782b9519c8" containerName="istio-proxy"
Jul 18 17:15:46.439149 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.439073    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce5a6238-5faa-4620-8182-98e628c6d47c" containerName="cleanup"
Jul 18 17:15:46.442197 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.442177    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.444960 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.444940    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-189241-openshift-default-dockercfg-6s4bf\""
Jul 18 17:15:46.445097 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.445080    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 18 17:15:46.454303 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.454281    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"]
Jul 18 17:15:46.467162 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.467138    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:46.504015 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.503984    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istiod-ca-cert\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.504164 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.504029    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-547rj\" (UniqueName: \"kubernetes.io/projected/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-kube-api-access-547rj\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.504164 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.504055    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-workload-certs\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.504274 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.504170    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-data\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.504274 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.504200    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-token\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.504274 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.504221    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-envoy\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.504429 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.504298    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-credential-socket\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.504429 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.504346    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-workload-socket\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.504429 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.504385    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-podinfo\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.595415 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.595379    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:46.605738 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.605711    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-data\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.605903 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.605746    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-token\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.605903 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.605771    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-envoy\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.606021 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.605897    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-credential-socket\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.606021 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.605933    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-workload-socket\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.606021 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.605963    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-podinfo\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.606021 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.605984    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istiod-ca-cert\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.606219 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.606037    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-547rj\" (UniqueName: \"kubernetes.io/projected/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-kube-api-access-547rj\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.606219 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.606080    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-workload-certs\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.606219 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.606191    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-data\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.606378 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.606283    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-credential-socket\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.606378 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.606367    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-workload-socket\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.606511 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.606485    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-workload-certs\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.606773 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.606746    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istiod-ca-cert\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.608196 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.608178    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-envoy\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.608417 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.608401    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-podinfo\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.614389 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.614358    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-token\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.614510 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.614493    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-547rj\" (UniqueName: \"kubernetes.io/projected/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-kube-api-access-547rj\") pod \"e2e-inf-a-189241-openshift-default-5f69547847-v5v6w\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") " pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.752752 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.752665    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:46.894501 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.894467    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"]
Jul 18 17:15:46.897530 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:15:46.897498    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb88a399_6a17_4fd8_b709_60bc11c5aa3a.slice/crio-277d5bb37c3bcefbf6e9a69cd76c5d0ceee6b1aa1b68b4bc13797b8a18c735ed WatchSource:0}: Error finding container 277d5bb37c3bcefbf6e9a69cd76c5d0ceee6b1aa1b68b4bc13797b8a18c735ed: Status 404 returned error can't find the container with id 277d5bb37c3bcefbf6e9a69cd76c5d0ceee6b1aa1b68b4bc13797b8a18c735ed
Jul 18 17:15:46.899581 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.899543    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:15:46.899659 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.899620    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:15:46.899659 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:46.899649    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:15:47.745550 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:47.745513    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w" event={"ID":"bb88a399-6a17-4fd8-b709-60bc11c5aa3a","Type":"ContainerStarted","Data":"efc2cfa5b9bcbb4c8446abede87b6b73fb3db2fe138c534dc4eeacbca280920e"}
Jul 18 17:15:47.745730 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:47.745571    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w" event={"ID":"bb88a399-6a17-4fd8-b709-60bc11c5aa3a","Type":"ContainerStarted","Data":"277d5bb37c3bcefbf6e9a69cd76c5d0ceee6b1aa1b68b4bc13797b8a18c735ed"}
Jul 18 17:15:47.752836 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:47.752808    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:47.758337 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:47.758312    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:47.767766 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:47.767723    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w" podStartSLOduration=1.7677076189999998 podStartE2EDuration="1.767707619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:15:47.766604923 +0000 UTC m=+2366.464435644" watchObservedRunningTime="2026-07-18 17:15:47.767707619 +0000 UTC m=+2366.465538303"
Jul 18 17:15:48.749628 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:48.749602    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:48.750604 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:48.750588    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:15:49.701080 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:49.701042    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:49.773485 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:49.773445    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"]
Jul 18 17:15:49.777101 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:49.777081    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"
Jul 18 17:15:49.779730 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:49.779705    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-189241-serving-cert\""
Jul 18 17:15:49.779893 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:49.779760    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 18 17:15:49.779893 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:49.779795    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:49.780895 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:49.780877    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 18 17:15:49.780964 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:49.780882    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2pqsv\""
Jul 18 17:15:49.788924 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:49.788902    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"]
Jul 18 17:15:49.848579 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:49.848552    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:49.935792 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:49.935759    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9500fe14-ce86-47bd-bbfc-253b88da862c-maas-api-tls\") pod \"maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd\" (UID: \"9500fe14-ce86-47bd-bbfc-253b88da862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"
Jul 18 17:15:49.936021 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:49.935808    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9500fe14-ce86-47bd-bbfc-253b88da862c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd\" (UID: \"9500fe14-ce86-47bd-bbfc-253b88da862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"
Jul 18 17:15:49.936021 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:49.935914    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvsx\" (UniqueName: \"kubernetes.io/projected/9500fe14-ce86-47bd-bbfc-253b88da862c-kube-api-access-swvsx\") pod \"maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd\" (UID: \"9500fe14-ce86-47bd-bbfc-253b88da862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"
Jul 18 17:15:50.035458 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:50.035366    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:50.036524 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:50.036503    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9500fe14-ce86-47bd-bbfc-253b88da862c-maas-api-tls\") pod \"maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd\" (UID: \"9500fe14-ce86-47bd-bbfc-253b88da862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"
Jul 18 17:15:50.036585 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:50.036547    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9500fe14-ce86-47bd-bbfc-253b88da862c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd\" (UID: \"9500fe14-ce86-47bd-bbfc-253b88da862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"
Jul 18 17:15:50.036585 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:50.036573    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvsx\" (UniqueName: \"kubernetes.io/projected/9500fe14-ce86-47bd-bbfc-253b88da862c-kube-api-access-swvsx\") pod \"maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd\" (UID: \"9500fe14-ce86-47bd-bbfc-253b88da862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"
Jul 18 17:15:50.037203 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:50.037180    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9500fe14-ce86-47bd-bbfc-253b88da862c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd\" (UID: \"9500fe14-ce86-47bd-bbfc-253b88da862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"
Jul 18 17:15:50.039190 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:50.039167    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9500fe14-ce86-47bd-bbfc-253b88da862c-maas-api-tls\") pod \"maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd\" (UID: \"9500fe14-ce86-47bd-bbfc-253b88da862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"
Jul 18 17:15:50.045755 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:50.045733    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvsx\" (UniqueName: \"kubernetes.io/projected/9500fe14-ce86-47bd-bbfc-253b88da862c-kube-api-access-swvsx\") pod \"maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd\" (UID: \"9500fe14-ce86-47bd-bbfc-253b88da862c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"
Jul 18 17:15:50.088884 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:50.088830    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"
Jul 18 17:15:50.225895 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:50.225830    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"]
Jul 18 17:15:50.227202 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:15:50.227174    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9500fe14_ce86_47bd_bbfc_253b88da862c.slice/crio-0abd3e70a5bdc5a19e5abb4954d243ebb85e6b5f91e87638715707a86562acb4 WatchSource:0}: Error finding container 0abd3e70a5bdc5a19e5abb4954d243ebb85e6b5f91e87638715707a86562acb4: Status 404 returned error can't find the container with id 0abd3e70a5bdc5a19e5abb4954d243ebb85e6b5f91e87638715707a86562acb4
Jul 18 17:15:50.271568 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:50.271528    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:50.372570 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:50.372543    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:50.759667 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:50.759580    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd" event={"ID":"9500fe14-ce86-47bd-bbfc-253b88da862c","Type":"ContainerStarted","Data":"0abd3e70a5bdc5a19e5abb4954d243ebb85e6b5f91e87638715707a86562acb4"}
Jul 18 17:15:51.766093 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:51.766058    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd" event={"ID":"9500fe14-ce86-47bd-bbfc-253b88da862c","Type":"ContainerStarted","Data":"3d29339ebc38a59ff7e821854fed3efe8bb5d3b5ba883f67f87adddd85928915"}
Jul 18 17:15:51.766482 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:51.766111    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"
Jul 18 17:15:51.783726 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:51.783681    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd" podStartSLOduration=2.244658655 podStartE2EDuration="2.783665555s" totalImagesPullingTime="539.0069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:15:51.783113959 +0000 UTC m=+2370.480944655" watchObservedRunningTime="2026-07-18 17:15:51.783665555 +0000 UTC m=+2370.481496238"
Jul 18 17:15:54.160253 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:54.160202    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:54.497072 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:54.496983    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:54.802739 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:54.802641    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:55.079668 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:55.079635    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:57.500455 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:57.500420    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:57.597391 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:57.597335    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:57.782385 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:57.782300    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"
Jul 18 17:15:57.788870 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:57.787203    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"]
Jul 18 17:15:57.795446 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:57.794835    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:15:57.799871 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:57.798589    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4554ed-serving-cert\""
Jul 18 17:15:57.810212 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:57.810188    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"]
Jul 18 17:15:57.890341 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:57.890302    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:57.908405 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:57.908372    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd1fc7b-4590-44af-b318-7ad67af1a410-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj\" (UID: \"4fd1fc7b-4590-44af-b318-7ad67af1a410\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:15:57.908592 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:57.908463    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd1fc7b-4590-44af-b318-7ad67af1a410-maas-api-tls\") pod \"maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj\" (UID: \"4fd1fc7b-4590-44af-b318-7ad67af1a410\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:15:57.908592 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:57.908482    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fssgz\" (UniqueName: \"kubernetes.io/projected/4fd1fc7b-4590-44af-b318-7ad67af1a410-kube-api-access-fssgz\") pod \"maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj\" (UID: \"4fd1fc7b-4590-44af-b318-7ad67af1a410\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:15:58.009813 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:58.009772    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd1fc7b-4590-44af-b318-7ad67af1a410-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj\" (UID: \"4fd1fc7b-4590-44af-b318-7ad67af1a410\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:15:58.010060 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:58.009911    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd1fc7b-4590-44af-b318-7ad67af1a410-maas-api-tls\") pod \"maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj\" (UID: \"4fd1fc7b-4590-44af-b318-7ad67af1a410\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:15:58.010060 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:58.009939    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fssgz\" (UniqueName: \"kubernetes.io/projected/4fd1fc7b-4590-44af-b318-7ad67af1a410-kube-api-access-fssgz\") pod \"maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj\" (UID: \"4fd1fc7b-4590-44af-b318-7ad67af1a410\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:15:58.010182 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:15:58.010063    2554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-serving-cert: secret "maas-api-e2e-inf-b-4554ed-serving-cert" not found
Jul 18 17:15:58.010182 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:15:58.010146    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd1fc7b-4590-44af-b318-7ad67af1a410-maas-api-tls podName:4fd1fc7b-4590-44af-b318-7ad67af1a410 nodeName:}" failed. No retries permitted until 2026-07-18 17:15:58.510122846 +0000 UTC m=+2377.207953508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4fd1fc7b-4590-44af-b318-7ad67af1a410-maas-api-tls") pod "maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj" (UID: "4fd1fc7b-4590-44af-b318-7ad67af1a410") : secret "maas-api-e2e-inf-b-4554ed-serving-cert" not found
Jul 18 17:15:58.010568 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:58.010549    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd1fc7b-4590-44af-b318-7ad67af1a410-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj\" (UID: \"4fd1fc7b-4590-44af-b318-7ad67af1a410\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:15:58.020542 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:58.020512    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fssgz\" (UniqueName: \"kubernetes.io/projected/4fd1fc7b-4590-44af-b318-7ad67af1a410-kube-api-access-fssgz\") pod \"maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj\" (UID: \"4fd1fc7b-4590-44af-b318-7ad67af1a410\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:15:58.060514 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:58.060447    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:58.351551 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:58.351524    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:58.456531 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:58.456500    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:15:58.514970 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:58.514919    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd1fc7b-4590-44af-b318-7ad67af1a410-maas-api-tls\") pod \"maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj\" (UID: \"4fd1fc7b-4590-44af-b318-7ad67af1a410\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:15:58.517562 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:58.517534    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd1fc7b-4590-44af-b318-7ad67af1a410-maas-api-tls\") pod \"maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj\" (UID: \"4fd1fc7b-4590-44af-b318-7ad67af1a410\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:15:58.710271 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:58.710173    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:15:58.889329 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:58.888931    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"]
Jul 18 17:15:58.890498 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:15:58.890466    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd1fc7b_4590_44af_b318_7ad67af1a410.slice/crio-1c30b04fe3605bf2ce87d41f2605222d73f59f76541eda80aeda44bcdfe5ad0b WatchSource:0}: Error finding container 1c30b04fe3605bf2ce87d41f2605222d73f59f76541eda80aeda44bcdfe5ad0b: Status 404 returned error can't find the container with id 1c30b04fe3605bf2ce87d41f2605222d73f59f76541eda80aeda44bcdfe5ad0b
Jul 18 17:15:59.811689 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:59.811646    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj" event={"ID":"4fd1fc7b-4590-44af-b318-7ad67af1a410","Type":"ContainerStarted","Data":"ad079a210a361285940d8060e97d90b366abdbac3f30b22715210d58d54740d8"}
Jul 18 17:15:59.812232 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:59.811703    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj" event={"ID":"4fd1fc7b-4590-44af-b318-7ad67af1a410","Type":"ContainerStarted","Data":"1c30b04fe3605bf2ce87d41f2605222d73f59f76541eda80aeda44bcdfe5ad0b"}
Jul 18 17:15:59.812232 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:59.811773    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:15:59.829955 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:15:59.829903    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj" podStartSLOduration=2.466962134 podStartE2EDuration="2.829885943s" totalImagesPullingTime="362.923809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:15:59.828392095 +0000 UTC m=+2378.526222779" watchObservedRunningTime="2026-07-18 17:15:59.829885943 +0000 UTC m=+2378.527716626"
Jul 18 17:16:01.978716 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:01.978677    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:16:03.750821 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:03.750783    2554 patch_prober.go:28] interesting pod/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 18 17:16:03.751289 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:03.750871    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w" podUID="bb88a399-6a17-4fd8-b709-60bc11c5aa3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 18 17:16:03.831145 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:03.831112    2554 generic.go:356] "Generic (PLEG): container finished" podID="bb88a399-6a17-4fd8-b709-60bc11c5aa3a" containerID="efc2cfa5b9bcbb4c8446abede87b6b73fb3db2fe138c534dc4eeacbca280920e" exitCode=0
Jul 18 17:16:03.831316 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:03.831166    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w" event={"ID":"bb88a399-6a17-4fd8-b709-60bc11c5aa3a","Type":"ContainerDied","Data":"efc2cfa5b9bcbb4c8446abede87b6b73fb3db2fe138c534dc4eeacbca280920e"}
Jul 18 17:16:03.831546 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:03.831532    2554 scope.go:117] "RemoveContainer" containerID="efc2cfa5b9bcbb4c8446abede87b6b73fb3db2fe138c534dc4eeacbca280920e"
Jul 18 17:16:03.832573 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:03.832540    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:16:03.832646 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:03.832629    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:16:03.832695 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:03.832680    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:16:04.491592 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.491541    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"]
Jul 18 17:16:04.494992 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.494967    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.497859 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.497831    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-189241\"/\"openshift-service-ca.crt\""
Jul 18 17:16:04.499127 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.499105    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-189241\"/\"default-dockercfg-qn2nc\""
Jul 18 17:16:04.499253 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.499127    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-189241\"/\"test-model-189241-kserve-self-signed-certs\""
Jul 18 17:16:04.499253 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.499208    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-189241\"/\"kube-root-ca.crt\""
Jul 18 17:16:04.504546 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.504523    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"]
Jul 18 17:16:04.528976 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.528934    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:16:04.675063 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.675029    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-home\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.675063 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.675064    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-dshm\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.675276 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.675089    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-model-cache\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.675276 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.675117    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ca8da9-7690-4330-af69-c81a92eb3e23-tls-certs\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.675276 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.675182    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vltgj\" (UniqueName: \"kubernetes.io/projected/98ca8da9-7690-4330-af69-c81a92eb3e23-kube-api-access-vltgj\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.675276 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.675235    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-tmp-dir\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.777310 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.776738    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-home\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.777310 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.776840    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-dshm\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.777310 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.776919    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-model-cache\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.777310 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.776953    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ca8da9-7690-4330-af69-c81a92eb3e23-tls-certs\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.777310 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.776990    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vltgj\" (UniqueName: \"kubernetes.io/projected/98ca8da9-7690-4330-af69-c81a92eb3e23-kube-api-access-vltgj\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.777310 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.777038    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-tmp-dir\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.777310 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.777309    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-home\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.778141 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.777386    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-tmp-dir\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.778141 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.777631    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-model-cache\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.781527 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.781498    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ca8da9-7690-4330-af69-c81a92eb3e23-tls-certs\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.789185 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.789115    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-dshm\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.792230 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.792190    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vltgj\" (UniqueName: \"kubernetes.io/projected/98ca8da9-7690-4330-af69-c81a92eb3e23-kube-api-access-vltgj\") pod \"test-model-189241-kserve-7fc49d7659-plwnr\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") " pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.805837 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.805806    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:04.837695 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.837659    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w" event={"ID":"bb88a399-6a17-4fd8-b709-60bc11c5aa3a","Type":"ContainerStarted","Data":"77529e1851428de77ddddd183313cc1e965c4bfb123377a3d1c91485ca896b67"}
Jul 18 17:16:04.972195 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:04.972115    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"]
Jul 18 17:16:04.993540 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:16:04.993495    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98ca8da9_7690_4330_af69_c81a92eb3e23.slice/crio-42b8981e0b9981e66eb0aac953048297fd96437cd2414782eac4bf200117d79d WatchSource:0}: Error finding container 42b8981e0b9981e66eb0aac953048297fd96437cd2414782eac4bf200117d79d: Status 404 returned error can't find the container with id 42b8981e0b9981e66eb0aac953048297fd96437cd2414782eac4bf200117d79d
Jul 18 17:16:05.752906 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:05.752866    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:16:05.753135 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:05.753112    2554 patch_prober.go:28] interesting pod/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 18 17:16:05.753219 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:05.753167    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w" podUID="bb88a399-6a17-4fd8-b709-60bc11c5aa3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 18 17:16:05.822128 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:05.822102    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:16:05.843833 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:05.843800    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr" event={"ID":"98ca8da9-7690-4330-af69-c81a92eb3e23","Type":"ContainerStarted","Data":"f495264521447bdde8e38263d6da2af428f4736e8068d5730a06c7f7843c1c01"}
Jul 18 17:16:05.844030 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:05.843870    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr" event={"ID":"98ca8da9-7690-4330-af69-c81a92eb3e23","Type":"ContainerStarted","Data":"42b8981e0b9981e66eb0aac953048297fd96437cd2414782eac4bf200117d79d"}
Jul 18 17:16:05.861225 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:05.861178    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr" podStartSLOduration=1.861164127 podStartE2EDuration="1.861164127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:16:05.85933827 +0000 UTC m=+2384.557168955" watchObservedRunningTime="2026-07-18 17:16:05.861164127 +0000 UTC m=+2384.558994810"
Jul 18 17:16:06.753503 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:06.753463    2554 patch_prober.go:28] interesting pod/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 18 17:16:06.753690 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:06.753524    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w" podUID="bb88a399-6a17-4fd8-b709-60bc11c5aa3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 18 17:16:07.753058 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:07.753017    2554 patch_prober.go:28] interesting pod/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 18 17:16:07.753426 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:07.753085    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w" podUID="bb88a399-6a17-4fd8-b709-60bc11c5aa3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 18 17:16:08.753409 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:08.753371    2554 patch_prober.go:28] interesting pod/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 18 17:16:08.753786 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:08.753426    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w" podUID="bb88a399-6a17-4fd8-b709-60bc11c5aa3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 18 17:16:09.757441 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:09.757411    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:16:09.757857 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:09.757822    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:16:09.758450 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:09.758432    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:16:12.448991 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:12.448955    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:16:12.548600 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:12.548564    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:16:14.806665 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:14.806627    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:14.807124 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:14.806679    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:14.819184 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:14.819159    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:14.898652 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:14.898618    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:16:19.207225 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:19.207186    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:16:19.307236 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:19.307193    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:16:21.990541 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:21.990508    2554 scope.go:117] "RemoveContainer" containerID="e25776780660312a09b5453acc1ad68a6e5c3ad8fdaaf57bf8a95051abef08a5"
Jul 18 17:16:21.998578 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:21.998560    2554 scope.go:117] "RemoveContainer" containerID="206b2639aa2e52cbc8e9c20bdc9fd4bc67a76119ab44fdcbd875fe2aaec3f03d"
Jul 18 17:16:40.071270 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.071234    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"]
Jul 18 17:16:40.074758 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.074739    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.078690 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.078669    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4554ed\"/\"test-model-4554ed-kserve-self-signed-certs\""
Jul 18 17:16:40.078797 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.078702    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4554ed\"/\"openshift-service-ca.crt\""
Jul 18 17:16:40.078797 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.078671    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4554ed\"/\"default-dockercfg-wfv58\""
Jul 18 17:16:40.078797 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.078671    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4554ed\"/\"kube-root-ca.crt\""
Jul 18 17:16:40.082169 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.082145    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"]
Jul 18 17:16:40.105000 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.104912    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:16:40.213056 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.213025    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-home\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.213242 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.213119    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-model-cache\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.213242 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.213149    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-tmp-dir\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.213242 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.213173    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42ab2dba-dea6-4495-a4cc-ea92993af530-tls-certs\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.213242 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.213238    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-dshm\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.213391 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.213316    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj579\" (UniqueName: \"kubernetes.io/projected/42ab2dba-dea6-4495-a4cc-ea92993af530-kube-api-access-pj579\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.314339 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.314299    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-home\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.314523 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.314366    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-model-cache\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.314523 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.314383    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-tmp-dir\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.314523 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.314403    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42ab2dba-dea6-4495-a4cc-ea92993af530-tls-certs\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.314523 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.314429    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-dshm\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.314523 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.314477    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj579\" (UniqueName: \"kubernetes.io/projected/42ab2dba-dea6-4495-a4cc-ea92993af530-kube-api-access-pj579\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.314794 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.314752    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-home\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.314839 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.314794    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-tmp-dir\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.314839 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.314809    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-model-cache\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.316772 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.316751    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-dshm\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.316927 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.316911    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42ab2dba-dea6-4495-a4cc-ea92993af530-tls-certs\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.322376 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.322316    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj579\" (UniqueName: \"kubernetes.io/projected/42ab2dba-dea6-4495-a4cc-ea92993af530-kube-api-access-pj579\") pod \"test-model-4554ed-kserve-759fb5789f-bljhd\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") " pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.385115 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.385074    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:40.522532 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:40.522505    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"]
Jul 18 17:16:40.525188 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:16:40.525147    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42ab2dba_dea6_4495_a4cc_ea92993af530.slice/crio-0198a5dbf8cf23c99e4970170e08fa1200683d38d8d83272aaa55dfc172295c1 WatchSource:0}: Error finding container 0198a5dbf8cf23c99e4970170e08fa1200683d38d8d83272aaa55dfc172295c1: Status 404 returned error can't find the container with id 0198a5dbf8cf23c99e4970170e08fa1200683d38d8d83272aaa55dfc172295c1
Jul 18 17:16:41.008355 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:41.008317    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd" event={"ID":"42ab2dba-dea6-4495-a4cc-ea92993af530","Type":"ContainerStarted","Data":"f0d655568a49823c7fb624aea5f8828bc9346c72f1f2c6ddd5e62f2ee103a76d"}
Jul 18 17:16:41.008355 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:41.008361    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd" event={"ID":"42ab2dba-dea6-4495-a4cc-ea92993af530","Type":"ContainerStarted","Data":"0198a5dbf8cf23c99e4970170e08fa1200683d38d8d83272aaa55dfc172295c1"}
Jul 18 17:16:41.026544 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:41.026488    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd" podStartSLOduration=1.02647396 podStartE2EDuration="1.02647396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:16:41.024766728 +0000 UTC m=+2419.722597467" watchObservedRunningTime="2026-07-18 17:16:41.02647396 +0000 UTC m=+2419.724304645"
Jul 18 17:16:50.385405 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:50.385326    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:50.385858 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:50.385494    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:50.403136 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:50.403106    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:16:51.073563 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:16:51.073527    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:17:11.114687 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:11.114640    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"]
Jul 18 17:17:11.136689 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:11.136657    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:11.983130 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:11.983096    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"]
Jul 18 17:17:12.007211 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:12.007174    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:12.634617 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:12.634582    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:12.735770 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:12.735731    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:22.524473 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:22.524434    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"]
Jul 18 17:17:22.524931 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:22.524674    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd" podUID="9500fe14-ce86-47bd-bbfc-253b88da862c" containerName="maas-api" containerID="cri-o://3d29339ebc38a59ff7e821854fed3efe8bb5d3b5ba883f67f87adddd85928915" gracePeriod=30
Jul 18 17:17:22.548977 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:22.548947    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:22.771255 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:22.771209    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd" podUID="9500fe14-ce86-47bd-bbfc-253b88da862c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.49:8443/health\": dial tcp 10.133.0.49:8443: connect: connection refused"
Jul 18 17:17:22.973838 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:22.973811    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"
Jul 18 17:17:23.117120 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.117087    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swvsx\" (UniqueName: \"kubernetes.io/projected/9500fe14-ce86-47bd-bbfc-253b88da862c-kube-api-access-swvsx\") pod \"9500fe14-ce86-47bd-bbfc-253b88da862c\" (UID: \"9500fe14-ce86-47bd-bbfc-253b88da862c\") "
Jul 18 17:17:23.117319 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.117132    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9500fe14-ce86-47bd-bbfc-253b88da862c-openshift-service-ca\") pod \"9500fe14-ce86-47bd-bbfc-253b88da862c\" (UID: \"9500fe14-ce86-47bd-bbfc-253b88da862c\") "
Jul 18 17:17:23.117319 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.117254    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9500fe14-ce86-47bd-bbfc-253b88da862c-maas-api-tls\") pod \"9500fe14-ce86-47bd-bbfc-253b88da862c\" (UID: \"9500fe14-ce86-47bd-bbfc-253b88da862c\") "
Jul 18 17:17:23.117494 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.117461    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9500fe14-ce86-47bd-bbfc-253b88da862c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9500fe14-ce86-47bd-bbfc-253b88da862c" (UID: "9500fe14-ce86-47bd-bbfc-253b88da862c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:17:23.119426 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.119386    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9500fe14-ce86-47bd-bbfc-253b88da862c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9500fe14-ce86-47bd-bbfc-253b88da862c" (UID: "9500fe14-ce86-47bd-bbfc-253b88da862c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:17:23.119631 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.119591    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9500fe14-ce86-47bd-bbfc-253b88da862c-kube-api-access-swvsx" (OuterVolumeSpecName: "kube-api-access-swvsx") pod "9500fe14-ce86-47bd-bbfc-253b88da862c" (UID: "9500fe14-ce86-47bd-bbfc-253b88da862c"). InnerVolumeSpecName "kube-api-access-swvsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:17:23.218353 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.218322    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swvsx\" (UniqueName: \"kubernetes.io/projected/9500fe14-ce86-47bd-bbfc-253b88da862c-kube-api-access-swvsx\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:23.218353 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.218349    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9500fe14-ce86-47bd-bbfc-253b88da862c-openshift-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:23.218353 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.218360    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9500fe14-ce86-47bd-bbfc-253b88da862c-maas-api-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:23.218922 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.218893    2554 generic.go:356] "Generic (PLEG): container finished" podID="9500fe14-ce86-47bd-bbfc-253b88da862c" containerID="3d29339ebc38a59ff7e821854fed3efe8bb5d3b5ba883f67f87adddd85928915" exitCode=0
Jul 18 17:17:23.219046 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.218951    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd" event={"ID":"9500fe14-ce86-47bd-bbfc-253b88da862c","Type":"ContainerDied","Data":"3d29339ebc38a59ff7e821854fed3efe8bb5d3b5ba883f67f87adddd85928915"}
Jul 18 17:17:23.219046 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.218955    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"
Jul 18 17:17:23.219046 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.218995    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd" event={"ID":"9500fe14-ce86-47bd-bbfc-253b88da862c","Type":"ContainerDied","Data":"0abd3e70a5bdc5a19e5abb4954d243ebb85e6b5f91e87638715707a86562acb4"}
Jul 18 17:17:23.219046 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.219016    2554 scope.go:117] "RemoveContainer" containerID="3d29339ebc38a59ff7e821854fed3efe8bb5d3b5ba883f67f87adddd85928915"
Jul 18 17:17:23.227881 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.227858    2554 scope.go:117] "RemoveContainer" containerID="3d29339ebc38a59ff7e821854fed3efe8bb5d3b5ba883f67f87adddd85928915"
Jul 18 17:17:23.228120 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:17:23.228102    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d29339ebc38a59ff7e821854fed3efe8bb5d3b5ba883f67f87adddd85928915\": container with ID starting with 3d29339ebc38a59ff7e821854fed3efe8bb5d3b5ba883f67f87adddd85928915 not found: ID does not exist" containerID="3d29339ebc38a59ff7e821854fed3efe8bb5d3b5ba883f67f87adddd85928915"
Jul 18 17:17:23.228172 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.228129    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d29339ebc38a59ff7e821854fed3efe8bb5d3b5ba883f67f87adddd85928915"} err="failed to get container status \"3d29339ebc38a59ff7e821854fed3efe8bb5d3b5ba883f67f87adddd85928915\": rpc error: code = NotFound desc = could not find container \"3d29339ebc38a59ff7e821854fed3efe8bb5d3b5ba883f67f87adddd85928915\": container with ID starting with 3d29339ebc38a59ff7e821854fed3efe8bb5d3b5ba883f67f87adddd85928915 not found: ID does not exist"
Jul 18 17:17:23.239360 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.239340    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"]
Jul 18 17:17:23.242958 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.242937    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-189241-5b7d6754f4-j8zjd"]
Jul 18 17:17:23.765368 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:23.765332    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9500fe14-ce86-47bd-bbfc-253b88da862c" path="/var/lib/kubelet/pods/9500fe14-ce86-47bd-bbfc-253b88da862c/volumes"
Jul 18 17:17:26.135054 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.135015    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr" podUID="98ca8da9-7690-4330-af69-c81a92eb3e23" containerName="main" containerID="cri-o://f495264521447bdde8e38263d6da2af428f4736e8068d5730a06c7f7843c1c01" gracePeriod=45
Jul 18 17:17:26.389251 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.389192    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:17:26.446592 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.446559    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-home\") pod \"98ca8da9-7690-4330-af69-c81a92eb3e23\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") "
Jul 18 17:17:26.446737 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.446641    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vltgj\" (UniqueName: \"kubernetes.io/projected/98ca8da9-7690-4330-af69-c81a92eb3e23-kube-api-access-vltgj\") pod \"98ca8da9-7690-4330-af69-c81a92eb3e23\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") "
Jul 18 17:17:26.446737 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.446667    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-tmp-dir\") pod \"98ca8da9-7690-4330-af69-c81a92eb3e23\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") "
Jul 18 17:17:26.446737 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.446691    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-dshm\") pod \"98ca8da9-7690-4330-af69-c81a92eb3e23\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") "
Jul 18 17:17:26.446737 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.446726    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-model-cache\") pod \"98ca8da9-7690-4330-af69-c81a92eb3e23\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") "
Jul 18 17:17:26.447015 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.446770    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ca8da9-7690-4330-af69-c81a92eb3e23-tls-certs\") pod \"98ca8da9-7690-4330-af69-c81a92eb3e23\" (UID: \"98ca8da9-7690-4330-af69-c81a92eb3e23\") "
Jul 18 17:17:26.447015 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.446791    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-home" (OuterVolumeSpecName: "home") pod "98ca8da9-7690-4330-af69-c81a92eb3e23" (UID: "98ca8da9-7690-4330-af69-c81a92eb3e23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:17:26.447015 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.446972    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98ca8da9-7690-4330-af69-c81a92eb3e23" (UID: "98ca8da9-7690-4330-af69-c81a92eb3e23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:17:26.447147 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.447044    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-model-cache" (OuterVolumeSpecName: "model-cache") pod "98ca8da9-7690-4330-af69-c81a92eb3e23" (UID: "98ca8da9-7690-4330-af69-c81a92eb3e23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:17:26.447147 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.447098    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-model-cache\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:26.447147 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.447119    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-home\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:26.447147 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.447136    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-tmp-dir\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:26.448980 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.448959    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-dshm" (OuterVolumeSpecName: "dshm") pod "98ca8da9-7690-4330-af69-c81a92eb3e23" (UID: "98ca8da9-7690-4330-af69-c81a92eb3e23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:17:26.449075 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.449049    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ca8da9-7690-4330-af69-c81a92eb3e23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98ca8da9-7690-4330-af69-c81a92eb3e23" (UID: "98ca8da9-7690-4330-af69-c81a92eb3e23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:17:26.449131 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.449113    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ca8da9-7690-4330-af69-c81a92eb3e23-kube-api-access-vltgj" (OuterVolumeSpecName: "kube-api-access-vltgj") pod "98ca8da9-7690-4330-af69-c81a92eb3e23" (UID: "98ca8da9-7690-4330-af69-c81a92eb3e23"). InnerVolumeSpecName "kube-api-access-vltgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:17:26.547682 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.547647    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98ca8da9-7690-4330-af69-c81a92eb3e23-dshm\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:26.547682 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.547677    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98ca8da9-7690-4330-af69-c81a92eb3e23-tls-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:26.547682 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:26.547691    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vltgj\" (UniqueName: \"kubernetes.io/projected/98ca8da9-7690-4330-af69-c81a92eb3e23-kube-api-access-vltgj\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:27.005371 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.005312    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd" podUID="42ab2dba-dea6-4495-a4cc-ea92993af530" containerName="main" containerID="cri-o://f0d655568a49823c7fb624aea5f8828bc9346c72f1f2c6ddd5e62f2ee103a76d" gracePeriod=45
Jul 18 17:17:27.203639 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.203605    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:27.241119 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.241094    2554 generic.go:356] "Generic (PLEG): container finished" podID="98ca8da9-7690-4330-af69-c81a92eb3e23" containerID="f495264521447bdde8e38263d6da2af428f4736e8068d5730a06c7f7843c1c01" exitCode=0
Jul 18 17:17:27.241243 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.241134    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"
Jul 18 17:17:27.241243 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.241186    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr" event={"ID":"98ca8da9-7690-4330-af69-c81a92eb3e23","Type":"ContainerDied","Data":"f495264521447bdde8e38263d6da2af428f4736e8068d5730a06c7f7843c1c01"}
Jul 18 17:17:27.241243 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.241237    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr" event={"ID":"98ca8da9-7690-4330-af69-c81a92eb3e23","Type":"ContainerDied","Data":"42b8981e0b9981e66eb0aac953048297fd96437cd2414782eac4bf200117d79d"}
Jul 18 17:17:27.241411 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.241260    2554 scope.go:117] "RemoveContainer" containerID="f495264521447bdde8e38263d6da2af428f4736e8068d5730a06c7f7843c1c01"
Jul 18 17:17:27.242575 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.242549    2554 generic.go:356] "Generic (PLEG): container finished" podID="42ab2dba-dea6-4495-a4cc-ea92993af530" containerID="f0d655568a49823c7fb624aea5f8828bc9346c72f1f2c6ddd5e62f2ee103a76d" exitCode=0
Jul 18 17:17:27.242656 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.242601    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd" event={"ID":"42ab2dba-dea6-4495-a4cc-ea92993af530","Type":"ContainerDied","Data":"f0d655568a49823c7fb624aea5f8828bc9346c72f1f2c6ddd5e62f2ee103a76d"}
Jul 18 17:17:27.259284 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.259265    2554 scope.go:117] "RemoveContainer" containerID="f495264521447bdde8e38263d6da2af428f4736e8068d5730a06c7f7843c1c01"
Jul 18 17:17:27.259555 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:17:27.259501    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f495264521447bdde8e38263d6da2af428f4736e8068d5730a06c7f7843c1c01\": container with ID starting with f495264521447bdde8e38263d6da2af428f4736e8068d5730a06c7f7843c1c01 not found: ID does not exist" containerID="f495264521447bdde8e38263d6da2af428f4736e8068d5730a06c7f7843c1c01"
Jul 18 17:17:27.259555 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.259536    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f495264521447bdde8e38263d6da2af428f4736e8068d5730a06c7f7843c1c01"} err="failed to get container status \"f495264521447bdde8e38263d6da2af428f4736e8068d5730a06c7f7843c1c01\": rpc error: code = NotFound desc = could not find container \"f495264521447bdde8e38263d6da2af428f4736e8068d5730a06c7f7843c1c01\": container with ID starting with f495264521447bdde8e38263d6da2af428f4736e8068d5730a06c7f7843c1c01 not found: ID does not exist"
Jul 18 17:17:27.263774 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.263755    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:17:27.279762 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.279739    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"]
Jul 18 17:17:27.284562 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.284543    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-189241/test-model-189241-kserve-7fc49d7659-plwnr"]
Jul 18 17:17:27.353099 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.353071    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj579\" (UniqueName: \"kubernetes.io/projected/42ab2dba-dea6-4495-a4cc-ea92993af530-kube-api-access-pj579\") pod \"42ab2dba-dea6-4495-a4cc-ea92993af530\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") "
Jul 18 17:17:27.353265 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.353129    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-home\") pod \"42ab2dba-dea6-4495-a4cc-ea92993af530\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") "
Jul 18 17:17:27.353265 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.353170    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42ab2dba-dea6-4495-a4cc-ea92993af530-tls-certs\") pod \"42ab2dba-dea6-4495-a4cc-ea92993af530\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") "
Jul 18 17:17:27.353265 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.353193    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-model-cache\") pod \"42ab2dba-dea6-4495-a4cc-ea92993af530\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") "
Jul 18 17:17:27.353265 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.353211    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-dshm\") pod \"42ab2dba-dea6-4495-a4cc-ea92993af530\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") "
Jul 18 17:17:27.353464 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.353281    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-tmp-dir\") pod \"42ab2dba-dea6-4495-a4cc-ea92993af530\" (UID: \"42ab2dba-dea6-4495-a4cc-ea92993af530\") "
Jul 18 17:17:27.353464 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.353361    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-home" (OuterVolumeSpecName: "home") pod "42ab2dba-dea6-4495-a4cc-ea92993af530" (UID: "42ab2dba-dea6-4495-a4cc-ea92993af530"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:17:27.353589 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.353470    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-model-cache" (OuterVolumeSpecName: "model-cache") pod "42ab2dba-dea6-4495-a4cc-ea92993af530" (UID: "42ab2dba-dea6-4495-a4cc-ea92993af530"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:17:27.353589 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.353570    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42ab2dba-dea6-4495-a4cc-ea92993af530" (UID: "42ab2dba-dea6-4495-a4cc-ea92993af530"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:17:27.353693 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.353661    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-home\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:27.353693 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.353679    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-model-cache\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:27.353776 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.353693    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-tmp-dir\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:27.355439 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.355399    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-dshm" (OuterVolumeSpecName: "dshm") pod "42ab2dba-dea6-4495-a4cc-ea92993af530" (UID: "42ab2dba-dea6-4495-a4cc-ea92993af530"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:17:27.355561 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.355493    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42ab2dba-dea6-4495-a4cc-ea92993af530-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42ab2dba-dea6-4495-a4cc-ea92993af530" (UID: "42ab2dba-dea6-4495-a4cc-ea92993af530"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:17:27.355561 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.355497    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ab2dba-dea6-4495-a4cc-ea92993af530-kube-api-access-pj579" (OuterVolumeSpecName: "kube-api-access-pj579") pod "42ab2dba-dea6-4495-a4cc-ea92993af530" (UID: "42ab2dba-dea6-4495-a4cc-ea92993af530"). InnerVolumeSpecName "kube-api-access-pj579". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:17:27.454806 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.454761    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj579\" (UniqueName: \"kubernetes.io/projected/42ab2dba-dea6-4495-a4cc-ea92993af530-kube-api-access-pj579\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:27.454806 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.454799    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42ab2dba-dea6-4495-a4cc-ea92993af530-tls-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:27.454806 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.454810    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42ab2dba-dea6-4495-a4cc-ea92993af530-dshm\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:27.600366 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.600338    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:27.767305 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:27.767272    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98ca8da9-7690-4330-af69-c81a92eb3e23" path="/var/lib/kubelet/pods/98ca8da9-7690-4330-af69-c81a92eb3e23/volumes"
Jul 18 17:17:28.249861 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:28.249814    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"
Jul 18 17:17:28.250312 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:28.249809    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd" event={"ID":"42ab2dba-dea6-4495-a4cc-ea92993af530","Type":"ContainerDied","Data":"0198a5dbf8cf23c99e4970170e08fa1200683d38d8d83272aaa55dfc172295c1"}
Jul 18 17:17:28.250312 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:28.249938    2554 scope.go:117] "RemoveContainer" containerID="f0d655568a49823c7fb624aea5f8828bc9346c72f1f2c6ddd5e62f2ee103a76d"
Jul 18 17:17:28.267381 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:28.267353    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"]
Jul 18 17:17:28.274758 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:28.274724    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4554ed/test-model-4554ed-kserve-759fb5789f-bljhd"]
Jul 18 17:17:29.765542 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:29.765509    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42ab2dba-dea6-4495-a4cc-ea92993af530" path="/var/lib/kubelet/pods/42ab2dba-dea6-4495-a4cc-ea92993af530/volumes"
Jul 18 17:17:33.404569 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:33.404531    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:33.464215 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:33.464162    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:33.662347 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:33.662259    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"]
Jul 18 17:17:33.662515 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:33.662489    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w" podUID="bb88a399-6a17-4fd8-b709-60bc11c5aa3a" containerName="istio-proxy" containerID="cri-o://77529e1851428de77ddddd183313cc1e965c4bfb123377a3d1c91485ca896b67" gracePeriod=30
Jul 18 17:17:33.679691 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:33.679669    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:33.885979 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:33.885947    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:34.016352 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.016256    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw"]
Jul 18 17:17:34.016697 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.016680    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ab2dba-dea6-4495-a4cc-ea92993af530" containerName="main"
Jul 18 17:17:34.016788 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.016699    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ab2dba-dea6-4495-a4cc-ea92993af530" containerName="main"
Jul 18 17:17:34.016788 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.016728    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98ca8da9-7690-4330-af69-c81a92eb3e23" containerName="main"
Jul 18 17:17:34.016788 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.016737    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="98ca8da9-7690-4330-af69-c81a92eb3e23" containerName="main"
Jul 18 17:17:34.016788 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.016756    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9500fe14-ce86-47bd-bbfc-253b88da862c" containerName="maas-api"
Jul 18 17:17:34.016788 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.016765    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9500fe14-ce86-47bd-bbfc-253b88da862c" containerName="maas-api"
Jul 18 17:17:34.017079 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.016839    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="42ab2dba-dea6-4495-a4cc-ea92993af530" containerName="main"
Jul 18 17:17:34.017079 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.016871    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9500fe14-ce86-47bd-bbfc-253b88da862c" containerName="maas-api"
Jul 18 17:17:34.017079 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.016884    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="98ca8da9-7690-4330-af69-c81a92eb3e23" containerName="main"
Jul 18 17:17:34.021264 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.021243    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw"
Jul 18 17:17:34.023700 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.023682    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7rq8\""
Jul 18 17:17:34.027173 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.027151    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw"]
Jul 18 17:17:34.116347 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.116309    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/498c7520-8d2e-4bf9-ad65-06f9900c042a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw\" (UID: \"498c7520-8d2e-4bf9-ad65-06f9900c042a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw"
Jul 18 17:17:34.217946 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.217907    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/498c7520-8d2e-4bf9-ad65-06f9900c042a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw\" (UID: \"498c7520-8d2e-4bf9-ad65-06f9900c042a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw"
Jul 18 17:17:34.218546 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.218525    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/498c7520-8d2e-4bf9-ad65-06f9900c042a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw\" (UID: \"498c7520-8d2e-4bf9-ad65-06f9900c042a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw"
Jul 18 17:17:34.333821 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.333789    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw"
Jul 18 17:17:34.473156 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.473129    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw"]
Jul 18 17:17:34.475739 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:17:34.475716    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498c7520_8d2e_4bf9_ad65_06f9900c042a.slice/crio-8e39831c61f9779b65d369500ec7da8e9c2f56ec472e2a87ed03d40322cc47f5 WatchSource:0}: Error finding container 8e39831c61f9779b65d369500ec7da8e9c2f56ec472e2a87ed03d40322cc47f5: Status 404 returned error can't find the container with id 8e39831c61f9779b65d369500ec7da8e9c2f56ec472e2a87ed03d40322cc47f5
Jul 18 17:17:34.526223 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.526190    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:34.628521 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:34.628426    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:35.281529 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:35.281496    2554 generic.go:356] "Generic (PLEG): container finished" podID="498c7520-8d2e-4bf9-ad65-06f9900c042a" containerID="4384894b7abd9bb9f078db5220223ba9fc99aba2b92ab42dc264fe47774c935c" exitCode=0
Jul 18 17:17:35.281686 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:35.281575    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw" event={"ID":"498c7520-8d2e-4bf9-ad65-06f9900c042a","Type":"ContainerDied","Data":"4384894b7abd9bb9f078db5220223ba9fc99aba2b92ab42dc264fe47774c935c"}
Jul 18 17:17:35.281686 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:35.281623    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw" event={"ID":"498c7520-8d2e-4bf9-ad65-06f9900c042a","Type":"ContainerStarted","Data":"8e39831c61f9779b65d369500ec7da8e9c2f56ec472e2a87ed03d40322cc47f5"}
Jul 18 17:17:36.417419 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:36.417396    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw"
Jul 18 17:17:36.542574 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:36.542480    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/498c7520-8d2e-4bf9-ad65-06f9900c042a-maas-api-service-ca\") pod \"498c7520-8d2e-4bf9-ad65-06f9900c042a\" (UID: \"498c7520-8d2e-4bf9-ad65-06f9900c042a\") "
Jul 18 17:17:36.542825 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:36.542802    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/498c7520-8d2e-4bf9-ad65-06f9900c042a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "498c7520-8d2e-4bf9-ad65-06f9900c042a" (UID: "498c7520-8d2e-4bf9-ad65-06f9900c042a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:17:36.643053 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:36.643018    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/498c7520-8d2e-4bf9-ad65-06f9900c042a-maas-api-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:37.292924 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:37.292886    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw" event={"ID":"498c7520-8d2e-4bf9-ad65-06f9900c042a","Type":"ContainerDied","Data":"8e39831c61f9779b65d369500ec7da8e9c2f56ec472e2a87ed03d40322cc47f5"}
Jul 18 17:17:37.293093 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:37.292940    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e39831c61f9779b65d369500ec7da8e9c2f56ec472e2a87ed03d40322cc47f5"
Jul 18 17:17:37.293093 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:37.292905    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw"
Jul 18 17:17:38.909358 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.909335    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:17:38.967047 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.966937    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-data\") pod \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") "
Jul 18 17:17:38.967047 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.966990    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-envoy\") pod \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") "
Jul 18 17:17:38.967047 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.967016    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-547rj\" (UniqueName: \"kubernetes.io/projected/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-kube-api-access-547rj\") pod \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") "
Jul 18 17:17:38.967047 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.967049    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-workload-certs\") pod \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") "
Jul 18 17:17:38.967388 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.967097    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-workload-socket\") pod \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") "
Jul 18 17:17:38.967388 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.967121    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-token\") pod \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") "
Jul 18 17:17:38.967388 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.967218    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istiod-ca-cert\") pod \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") "
Jul 18 17:17:38.967388 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.967268    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-podinfo\") pod \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") "
Jul 18 17:17:38.967388 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.967313    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-credential-socket\") pod \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\" (UID: \"bb88a399-6a17-4fd8-b709-60bc11c5aa3a\") "
Jul 18 17:17:38.967647 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.967461    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bb88a399-6a17-4fd8-b709-60bc11c5aa3a" (UID: "bb88a399-6a17-4fd8-b709-60bc11c5aa3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:17:38.967751 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.967642    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bb88a399-6a17-4fd8-b709-60bc11c5aa3a" (UID: "bb88a399-6a17-4fd8-b709-60bc11c5aa3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:17:38.967751 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.967653    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-workload-socket\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:38.967751 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.967665    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bb88a399-6a17-4fd8-b709-60bc11c5aa3a" (UID: "bb88a399-6a17-4fd8-b709-60bc11c5aa3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:17:38.967751 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.967679    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bb88a399-6a17-4fd8-b709-60bc11c5aa3a" (UID: "bb88a399-6a17-4fd8-b709-60bc11c5aa3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:17:38.968176 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.968147    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-data" (OuterVolumeSpecName: "istio-data") pod "bb88a399-6a17-4fd8-b709-60bc11c5aa3a" (UID: "bb88a399-6a17-4fd8-b709-60bc11c5aa3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:17:38.969489 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.969449    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bb88a399-6a17-4fd8-b709-60bc11c5aa3a" (UID: "bb88a399-6a17-4fd8-b709-60bc11c5aa3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:17:38.969489 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.969461    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-token" (OuterVolumeSpecName: "istio-token") pod "bb88a399-6a17-4fd8-b709-60bc11c5aa3a" (UID: "bb88a399-6a17-4fd8-b709-60bc11c5aa3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:17:38.969661 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.969557    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-kube-api-access-547rj" (OuterVolumeSpecName: "kube-api-access-547rj") pod "bb88a399-6a17-4fd8-b709-60bc11c5aa3a" (UID: "bb88a399-6a17-4fd8-b709-60bc11c5aa3a"). InnerVolumeSpecName "kube-api-access-547rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:17:38.969661 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:38.969651    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bb88a399-6a17-4fd8-b709-60bc11c5aa3a" (UID: "bb88a399-6a17-4fd8-b709-60bc11c5aa3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 17:17:39.068406 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.068362    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-data\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:39.068406 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.068400    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-envoy\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:39.068406 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.068414    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-547rj\" (UniqueName: \"kubernetes.io/projected/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-kube-api-access-547rj\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:39.068700 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.068427    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-workload-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:39.068700 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.068439    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-token\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:39.068700 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.068449    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istiod-ca-cert\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:39.068700 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.068460    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-istio-podinfo\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:39.068700 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.068470    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb88a399-6a17-4fd8-b709-60bc11c5aa3a-credential-socket\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:39.309701 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.309615    2554 generic.go:356] "Generic (PLEG): container finished" podID="bb88a399-6a17-4fd8-b709-60bc11c5aa3a" containerID="77529e1851428de77ddddd183313cc1e965c4bfb123377a3d1c91485ca896b67" exitCode=0
Jul 18 17:17:39.309701 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.309676    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"
Jul 18 17:17:39.309701 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.309688    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w" event={"ID":"bb88a399-6a17-4fd8-b709-60bc11c5aa3a","Type":"ContainerDied","Data":"77529e1851428de77ddddd183313cc1e965c4bfb123377a3d1c91485ca896b67"}
Jul 18 17:17:39.309984 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.309750    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w" event={"ID":"bb88a399-6a17-4fd8-b709-60bc11c5aa3a","Type":"ContainerDied","Data":"277d5bb37c3bcefbf6e9a69cd76c5d0ceee6b1aa1b68b4bc13797b8a18c735ed"}
Jul 18 17:17:39.309984 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.309772    2554 scope.go:117] "RemoveContainer" containerID="77529e1851428de77ddddd183313cc1e965c4bfb123377a3d1c91485ca896b67"
Jul 18 17:17:39.320626 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.320607    2554 scope.go:117] "RemoveContainer" containerID="efc2cfa5b9bcbb4c8446abede87b6b73fb3db2fe138c534dc4eeacbca280920e"
Jul 18 17:17:39.328589 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.328572    2554 scope.go:117] "RemoveContainer" containerID="77529e1851428de77ddddd183313cc1e965c4bfb123377a3d1c91485ca896b67"
Jul 18 17:17:39.328837 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:17:39.328818    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77529e1851428de77ddddd183313cc1e965c4bfb123377a3d1c91485ca896b67\": container with ID starting with 77529e1851428de77ddddd183313cc1e965c4bfb123377a3d1c91485ca896b67 not found: ID does not exist" containerID="77529e1851428de77ddddd183313cc1e965c4bfb123377a3d1c91485ca896b67"
Jul 18 17:17:39.328950 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.328865    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77529e1851428de77ddddd183313cc1e965c4bfb123377a3d1c91485ca896b67"} err="failed to get container status \"77529e1851428de77ddddd183313cc1e965c4bfb123377a3d1c91485ca896b67\": rpc error: code = NotFound desc = could not find container \"77529e1851428de77ddddd183313cc1e965c4bfb123377a3d1c91485ca896b67\": container with ID starting with 77529e1851428de77ddddd183313cc1e965c4bfb123377a3d1c91485ca896b67 not found: ID does not exist"
Jul 18 17:17:39.328950 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.328889    2554 scope.go:117] "RemoveContainer" containerID="efc2cfa5b9bcbb4c8446abede87b6b73fb3db2fe138c534dc4eeacbca280920e"
Jul 18 17:17:39.329164 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:17:39.329145    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc2cfa5b9bcbb4c8446abede87b6b73fb3db2fe138c534dc4eeacbca280920e\": container with ID starting with efc2cfa5b9bcbb4c8446abede87b6b73fb3db2fe138c534dc4eeacbca280920e not found: ID does not exist" containerID="efc2cfa5b9bcbb4c8446abede87b6b73fb3db2fe138c534dc4eeacbca280920e"
Jul 18 17:17:39.329212 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.329171    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc2cfa5b9bcbb4c8446abede87b6b73fb3db2fe138c534dc4eeacbca280920e"} err="failed to get container status \"efc2cfa5b9bcbb4c8446abede87b6b73fb3db2fe138c534dc4eeacbca280920e\": rpc error: code = NotFound desc = could not find container \"efc2cfa5b9bcbb4c8446abede87b6b73fb3db2fe138c534dc4eeacbca280920e\": container with ID starting with efc2cfa5b9bcbb4c8446abede87b6b73fb3db2fe138c534dc4eeacbca280920e not found: ID does not exist"
Jul 18 17:17:39.333672 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.333652    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"]
Jul 18 17:17:39.340338 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.340319    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-189241-openshift-default-5f69547847-v5v6w"]
Jul 18 17:17:39.765584 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:39.765550    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb88a399-6a17-4fd8-b709-60bc11c5aa3a" path="/var/lib/kubelet/pods/bb88a399-6a17-4fd8-b709-60bc11c5aa3a/volumes"
Jul 18 17:17:44.421402 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:44.421365    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"]
Jul 18 17:17:44.421918 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:44.421586    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj" podUID="4fd1fc7b-4590-44af-b318-7ad67af1a410" containerName="maas-api" containerID="cri-o://ad079a210a361285940d8060e97d90b366abdbac3f30b22715210d58d54740d8" gracePeriod=30
Jul 18 17:17:44.450591 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:44.450566    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:44.666965 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:44.666938    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:17:44.713336 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:44.713274    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd1fc7b-4590-44af-b318-7ad67af1a410-openshift-service-ca\") pod \"4fd1fc7b-4590-44af-b318-7ad67af1a410\" (UID: \"4fd1fc7b-4590-44af-b318-7ad67af1a410\") "
Jul 18 17:17:44.713336 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:44.713305    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fssgz\" (UniqueName: \"kubernetes.io/projected/4fd1fc7b-4590-44af-b318-7ad67af1a410-kube-api-access-fssgz\") pod \"4fd1fc7b-4590-44af-b318-7ad67af1a410\" (UID: \"4fd1fc7b-4590-44af-b318-7ad67af1a410\") "
Jul 18 17:17:44.713529 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:44.713337    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd1fc7b-4590-44af-b318-7ad67af1a410-maas-api-tls\") pod \"4fd1fc7b-4590-44af-b318-7ad67af1a410\" (UID: \"4fd1fc7b-4590-44af-b318-7ad67af1a410\") "
Jul 18 17:17:44.713689 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:44.713666    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd1fc7b-4590-44af-b318-7ad67af1a410-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4fd1fc7b-4590-44af-b318-7ad67af1a410" (UID: "4fd1fc7b-4590-44af-b318-7ad67af1a410"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:17:44.713786 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:44.713742    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd1fc7b-4590-44af-b318-7ad67af1a410-openshift-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:44.715666 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:44.715642    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd1fc7b-4590-44af-b318-7ad67af1a410-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4fd1fc7b-4590-44af-b318-7ad67af1a410" (UID: "4fd1fc7b-4590-44af-b318-7ad67af1a410"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:17:44.715666 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:44.715648    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd1fc7b-4590-44af-b318-7ad67af1a410-kube-api-access-fssgz" (OuterVolumeSpecName: "kube-api-access-fssgz") pod "4fd1fc7b-4590-44af-b318-7ad67af1a410" (UID: "4fd1fc7b-4590-44af-b318-7ad67af1a410"). InnerVolumeSpecName "kube-api-access-fssgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:17:44.814795 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:44.814763    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fssgz\" (UniqueName: \"kubernetes.io/projected/4fd1fc7b-4590-44af-b318-7ad67af1a410-kube-api-access-fssgz\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:44.814795 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:44.814790    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd1fc7b-4590-44af-b318-7ad67af1a410-maas-api-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:17:45.340706 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:45.340663    2554 generic.go:356] "Generic (PLEG): container finished" podID="4fd1fc7b-4590-44af-b318-7ad67af1a410" containerID="ad079a210a361285940d8060e97d90b366abdbac3f30b22715210d58d54740d8" exitCode=0
Jul 18 17:17:45.340901 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:45.340718    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"
Jul 18 17:17:45.340901 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:45.340742    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj" event={"ID":"4fd1fc7b-4590-44af-b318-7ad67af1a410","Type":"ContainerDied","Data":"ad079a210a361285940d8060e97d90b366abdbac3f30b22715210d58d54740d8"}
Jul 18 17:17:45.340901 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:45.340826    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj" event={"ID":"4fd1fc7b-4590-44af-b318-7ad67af1a410","Type":"ContainerDied","Data":"1c30b04fe3605bf2ce87d41f2605222d73f59f76541eda80aeda44bcdfe5ad0b"}
Jul 18 17:17:45.340901 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:45.340870    2554 scope.go:117] "RemoveContainer" containerID="ad079a210a361285940d8060e97d90b366abdbac3f30b22715210d58d54740d8"
Jul 18 17:17:45.350863 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:45.350828    2554 scope.go:117] "RemoveContainer" containerID="ad079a210a361285940d8060e97d90b366abdbac3f30b22715210d58d54740d8"
Jul 18 17:17:45.351158 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:17:45.351135    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad079a210a361285940d8060e97d90b366abdbac3f30b22715210d58d54740d8\": container with ID starting with ad079a210a361285940d8060e97d90b366abdbac3f30b22715210d58d54740d8 not found: ID does not exist" containerID="ad079a210a361285940d8060e97d90b366abdbac3f30b22715210d58d54740d8"
Jul 18 17:17:45.351209 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:45.351168    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad079a210a361285940d8060e97d90b366abdbac3f30b22715210d58d54740d8"} err="failed to get container status \"ad079a210a361285940d8060e97d90b366abdbac3f30b22715210d58d54740d8\": rpc error: code = NotFound desc = could not find container \"ad079a210a361285940d8060e97d90b366abdbac3f30b22715210d58d54740d8\": container with ID starting with ad079a210a361285940d8060e97d90b366abdbac3f30b22715210d58d54740d8 not found: ID does not exist"
Jul 18 17:17:45.361106 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:45.361081    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"]
Jul 18 17:17:45.364908 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:45.364889    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4554ed-855f575d75-fd4rj"]
Jul 18 17:17:45.766327 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:45.766250    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fd1fc7b-4590-44af-b318-7ad67af1a410" path="/var/lib/kubelet/pods/4fd1fc7b-4590-44af-b318-7ad67af1a410/volumes"
Jul 18 17:17:49.085141 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:49.085094    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw"]
Jul 18 17:17:49.085814 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:49.085788    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:49.087038 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:49.087018    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4554ed-rh8bw"]
Jul 18 17:17:49.483029 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:49.482944    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:49.766334 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:49.766252    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="498c7520-8d2e-4bf9-ad65-06f9900c042a" path="/var/lib/kubelet/pods/498c7520-8d2e-4bf9-ad65-06f9900c042a/volumes"
Jul 18 17:17:55.197712 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:55.197668    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:55.271215 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:55.271184    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:55.460045 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:55.459948    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:55.684982 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:55.684945    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:56.035443 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.035405    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:56.351619 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.351587    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"]
Jul 18 17:17:56.352051 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.351931    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb88a399-6a17-4fd8-b709-60bc11c5aa3a" containerName="istio-proxy"
Jul 18 17:17:56.352051 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.351943    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb88a399-6a17-4fd8-b709-60bc11c5aa3a" containerName="istio-proxy"
Jul 18 17:17:56.352051 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.351961    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="498c7520-8d2e-4bf9-ad65-06f9900c042a" containerName="revoke-keys"
Jul 18 17:17:56.352051 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.351969    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="498c7520-8d2e-4bf9-ad65-06f9900c042a" containerName="revoke-keys"
Jul 18 17:17:56.352051 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.351976    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd1fc7b-4590-44af-b318-7ad67af1a410" containerName="maas-api"
Jul 18 17:17:56.352051 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.351982    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd1fc7b-4590-44af-b318-7ad67af1a410" containerName="maas-api"
Jul 18 17:17:56.352051 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.351989    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb88a399-6a17-4fd8-b709-60bc11c5aa3a" containerName="istio-proxy"
Jul 18 17:17:56.352051 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.351993    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb88a399-6a17-4fd8-b709-60bc11c5aa3a" containerName="istio-proxy"
Jul 18 17:17:56.352051 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.352044    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb88a399-6a17-4fd8-b709-60bc11c5aa3a" containerName="istio-proxy"
Jul 18 17:17:56.352051 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.352052    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd1fc7b-4590-44af-b318-7ad67af1a410" containerName="maas-api"
Jul 18 17:17:56.352428 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.352060    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="498c7520-8d2e-4bf9-ad65-06f9900c042a" containerName="revoke-keys"
Jul 18 17:17:56.359460 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.359442    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.362094 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.362070    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 18 17:17:56.362219 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.362134    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-73d394-openshift-default-dockercfg-jqtx8\""
Jul 18 17:17:56.369992 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.369968    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"]
Jul 18 17:17:56.371502 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.371477    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:56.470691 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.470653    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:56.519840 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.519805    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-credential-socket\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.519995 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.519868    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx2ps\" (UniqueName: \"kubernetes.io/projected/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-kube-api-access-tx2ps\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.519995 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.519920    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-token\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.519995 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.519965    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istiod-ca-cert\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.519995 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.519989    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-envoy\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.520142 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.520051    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-workload-certs\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.520142 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.520072    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-podinfo\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.520142 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.520113    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-data\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.520142 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.520139    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-workload-socket\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.621076 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.620974    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-data\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.621076 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.621020    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-workload-socket\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.621076 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.621070    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-credential-socket\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.621377 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.621085    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx2ps\" (UniqueName: \"kubernetes.io/projected/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-kube-api-access-tx2ps\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.621377 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.621103    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-token\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.621377 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.621136    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istiod-ca-cert\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.621377 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.621284    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-envoy\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.621377 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.621354    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-workload-certs\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.621633 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.621380    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-podinfo\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.621633 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.621435    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-data\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.621633 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.621578    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-credential-socket\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.621786 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.621633    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-workload-socket\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.621786 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.621741    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-workload-certs\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.621894 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.621805    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istiod-ca-cert\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.623592 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.623571    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-envoy\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.623890 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.623869    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-podinfo\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.629131 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.629107    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-token\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.629388 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.629370    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx2ps\" (UniqueName: \"kubernetes.io/projected/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-kube-api-access-tx2ps\") pod \"e2e-api-a-73d394-openshift-default-568d759479-kqhjq\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") " pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.670555 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.670509    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:56.811820 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.811787    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"]
Jul 18 17:17:56.821917 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.821878    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:17:56.822033 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.821946    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:17:56.822033 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:56.821973    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:17:57.396829 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:57.396793    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq" event={"ID":"95b4ef95-fc98-4eb0-994e-9ba27b9545e6","Type":"ContainerStarted","Data":"e8515fc0d7b240f41bdc5463c0567fd2e60bbc29a8c3a88ad97896c7c4675cee"}
Jul 18 17:17:57.397347 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:57.396905    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq" event={"ID":"95b4ef95-fc98-4eb0-994e-9ba27b9545e6","Type":"ContainerStarted","Data":"f461d5dbc1b22feb9cdb06be7e1173971457d11f48f0934be4906e34891c2fb4"}
Jul 18 17:17:57.418377 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:57.418330    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq" podStartSLOduration=1.418315013 podStartE2EDuration="1.418315013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:17:57.417452561 +0000 UTC m=+2496.115283270" watchObservedRunningTime="2026-07-18 17:17:57.418315013 +0000 UTC m=+2496.116145696"
Jul 18 17:17:57.671057 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:57.670981    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:57.675803 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:57.675778    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:58.401517 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:58.401481    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:58.402428 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:58.402413    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:17:59.409715 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.409684    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:59.490131 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.490099    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"]
Jul 18 17:17:59.490496 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.490482    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb88a399-6a17-4fd8-b709-60bc11c5aa3a" containerName="istio-proxy"
Jul 18 17:17:59.492083 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.492059    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:59.493326 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.493310    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"
Jul 18 17:17:59.496095 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.496074    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-73d394-serving-cert\""
Jul 18 17:17:59.496260 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.496245    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2pqsv\""
Jul 18 17:17:59.496366 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.496347    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 18 17:17:59.497264 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.497250    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 18 17:17:59.505306 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.505287    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"]
Jul 18 17:17:59.573813 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.573778    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:59.650898 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.650868    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa57603-a622-4661-8089-506f303de554-openshift-service-ca\") pod \"maas-api-e2e-api-a-73d394-5898c4449b-c5c86\" (UID: \"3aa57603-a622-4661-8089-506f303de554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"
Jul 18 17:17:59.651062 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.650946    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aa57603-a622-4661-8089-506f303de554-maas-api-tls\") pod \"maas-api-e2e-api-a-73d394-5898c4449b-c5c86\" (UID: \"3aa57603-a622-4661-8089-506f303de554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"
Jul 18 17:17:59.651062 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.650973    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99756\" (UniqueName: \"kubernetes.io/projected/3aa57603-a622-4661-8089-506f303de554-kube-api-access-99756\") pod \"maas-api-e2e-api-a-73d394-5898c4449b-c5c86\" (UID: \"3aa57603-a622-4661-8089-506f303de554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"
Jul 18 17:17:59.675426 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.675340    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:17:59.751524 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.751486    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa57603-a622-4661-8089-506f303de554-openshift-service-ca\") pod \"maas-api-e2e-api-a-73d394-5898c4449b-c5c86\" (UID: \"3aa57603-a622-4661-8089-506f303de554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"
Jul 18 17:17:59.751665 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.751542    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aa57603-a622-4661-8089-506f303de554-maas-api-tls\") pod \"maas-api-e2e-api-a-73d394-5898c4449b-c5c86\" (UID: \"3aa57603-a622-4661-8089-506f303de554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"
Jul 18 17:17:59.751665 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.751573    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99756\" (UniqueName: \"kubernetes.io/projected/3aa57603-a622-4661-8089-506f303de554-kube-api-access-99756\") pod \"maas-api-e2e-api-a-73d394-5898c4449b-c5c86\" (UID: \"3aa57603-a622-4661-8089-506f303de554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"
Jul 18 17:17:59.752235 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.752212    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa57603-a622-4661-8089-506f303de554-openshift-service-ca\") pod \"maas-api-e2e-api-a-73d394-5898c4449b-c5c86\" (UID: \"3aa57603-a622-4661-8089-506f303de554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"
Jul 18 17:17:59.754299 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.754274    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aa57603-a622-4661-8089-506f303de554-maas-api-tls\") pod \"maas-api-e2e-api-a-73d394-5898c4449b-c5c86\" (UID: \"3aa57603-a622-4661-8089-506f303de554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"
Jul 18 17:17:59.760229 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.760207    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99756\" (UniqueName: \"kubernetes.io/projected/3aa57603-a622-4661-8089-506f303de554-kube-api-access-99756\") pod \"maas-api-e2e-api-a-73d394-5898c4449b-c5c86\" (UID: \"3aa57603-a622-4661-8089-506f303de554\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"
Jul 18 17:17:59.804684 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.804659    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"
Jul 18 17:17:59.955025 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:17:59.955001    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"]
Jul 18 17:17:59.957168 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:17:59.957137    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa57603_a622_4661_8089_506f303de554.slice/crio-aeb49e95f711618bd82766253ff61956dbf59e629dfc662f8bee3eaf6a55d422 WatchSource:0}: Error finding container aeb49e95f711618bd82766253ff61956dbf59e629dfc662f8bee3eaf6a55d422: Status 404 returned error can't find the container with id aeb49e95f711618bd82766253ff61956dbf59e629dfc662f8bee3eaf6a55d422
Jul 18 17:18:00.086905 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:00.086871    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:00.182045 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:00.182011    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:00.411524 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:00.411486    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86" event={"ID":"3aa57603-a622-4661-8089-506f303de554","Type":"ContainerStarted","Data":"aeb49e95f711618bd82766253ff61956dbf59e629dfc662f8bee3eaf6a55d422"}
Jul 18 17:18:01.417782 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:01.417749    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86" event={"ID":"3aa57603-a622-4661-8089-506f303de554","Type":"ContainerStarted","Data":"5d3e091da192b8bc8b31c32fe3a4ffd08a294d142a3ebdb43c49f14b0cbc7a29"}
Jul 18 17:18:01.418196 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:01.417884    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"
Jul 18 17:18:01.437894 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:01.437833    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86" podStartSLOduration=1.917377032 podStartE2EDuration="2.437815789s" totalImagesPullingTime="520.438757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:18:01.435873049 +0000 UTC m=+2500.133703733" watchObservedRunningTime="2026-07-18 17:18:01.437815789 +0000 UTC m=+2500.135646506"
Jul 18 17:18:03.811709 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:03.811672    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:04.278489 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:04.278457    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:04.574837 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:04.574759    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:04.671189 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:04.671148    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:07.144506 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.144466    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:07.233802 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.233761    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:07.234991 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.234972    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"]
Jul 18 17:18:07.238538 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.238520    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"
Jul 18 17:18:07.241033 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.241010    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-376c15-serving-cert\""
Jul 18 17:18:07.251062 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.251039    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"]
Jul 18 17:18:07.311705 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.311665    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:07.413533 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.413444    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:07.420515 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.420488    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea291f58-b23d-4d55-88de-d8d4682ba848-maas-api-tls\") pod \"maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb\" (UID: \"ea291f58-b23d-4d55-88de-d8d4682ba848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"
Jul 18 17:18:07.420670 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.420519    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxvb6\" (UniqueName: \"kubernetes.io/projected/ea291f58-b23d-4d55-88de-d8d4682ba848-kube-api-access-cxvb6\") pod \"maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb\" (UID: \"ea291f58-b23d-4d55-88de-d8d4682ba848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"
Jul 18 17:18:07.420670 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.420550    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea291f58-b23d-4d55-88de-d8d4682ba848-openshift-service-ca\") pod \"maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb\" (UID: \"ea291f58-b23d-4d55-88de-d8d4682ba848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"
Jul 18 17:18:07.427647 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.427622    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"
Jul 18 17:18:07.521589 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.521553    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea291f58-b23d-4d55-88de-d8d4682ba848-openshift-service-ca\") pod \"maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb\" (UID: \"ea291f58-b23d-4d55-88de-d8d4682ba848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"
Jul 18 17:18:07.521773 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.521639    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea291f58-b23d-4d55-88de-d8d4682ba848-maas-api-tls\") pod \"maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb\" (UID: \"ea291f58-b23d-4d55-88de-d8d4682ba848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"
Jul 18 17:18:07.521773 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.521660    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxvb6\" (UniqueName: \"kubernetes.io/projected/ea291f58-b23d-4d55-88de-d8d4682ba848-kube-api-access-cxvb6\") pod \"maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb\" (UID: \"ea291f58-b23d-4d55-88de-d8d4682ba848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"
Jul 18 17:18:07.522263 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.522238    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea291f58-b23d-4d55-88de-d8d4682ba848-openshift-service-ca\") pod \"maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb\" (UID: \"ea291f58-b23d-4d55-88de-d8d4682ba848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"
Jul 18 17:18:07.524281 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.524258    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea291f58-b23d-4d55-88de-d8d4682ba848-maas-api-tls\") pod \"maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb\" (UID: \"ea291f58-b23d-4d55-88de-d8d4682ba848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"
Jul 18 17:18:07.530024 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.530001    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxvb6\" (UniqueName: \"kubernetes.io/projected/ea291f58-b23d-4d55-88de-d8d4682ba848-kube-api-access-cxvb6\") pod \"maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb\" (UID: \"ea291f58-b23d-4d55-88de-d8d4682ba848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"
Jul 18 17:18:07.548867 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.548826    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"
Jul 18 17:18:07.757677 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.757650    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"]
Jul 18 17:18:07.759428 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:18:07.759400    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea291f58_b23d_4d55_88de_d8d4682ba848.slice/crio-4f06f53ab3fd40feec88ebbf82d993a755d0710d668c47e030ddfddd08ce2ecb WatchSource:0}: Error finding container 4f06f53ab3fd40feec88ebbf82d993a755d0710d668c47e030ddfddd08ce2ecb: Status 404 returned error can't find the container with id 4f06f53ab3fd40feec88ebbf82d993a755d0710d668c47e030ddfddd08ce2ecb
Jul 18 17:18:07.913428 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:07.913380    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:08.014096 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:08.013997    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:08.451854 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:08.451824    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb" event={"ID":"ea291f58-b23d-4d55-88de-d8d4682ba848","Type":"ContainerStarted","Data":"aedb16102bfe356d76c1af8af9f83b247f449e4d0022529aaf1358e65cb83867"}
Jul 18 17:18:08.452294 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:08.451891    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb" event={"ID":"ea291f58-b23d-4d55-88de-d8d4682ba848","Type":"ContainerStarted","Data":"4f06f53ab3fd40feec88ebbf82d993a755d0710d668c47e030ddfddd08ce2ecb"}
Jul 18 17:18:08.452294 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:08.452007    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"
Jul 18 17:18:08.470234 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:08.470167    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb" podStartSLOduration=1.028350382 podStartE2EDuration="1.470147773s" totalImagesPullingTime="441.797391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:18:08.467789103 +0000 UTC m=+2507.165619788" watchObservedRunningTime="2026-07-18 17:18:08.470147773 +0000 UTC m=+2507.167978459"
Jul 18 17:18:09.369603 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:09.369561    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:11.625056 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:11.625004    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:13.403667 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:13.403628    2554 patch_prober.go:28] interesting pod/e2e-api-a-73d394-openshift-default-568d759479-kqhjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 18 17:18:13.404108 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:13.403686    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq" podUID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 18 17:18:13.478573 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:13.478542    2554 generic.go:356] "Generic (PLEG): container finished" podID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" containerID="e8515fc0d7b240f41bdc5463c0567fd2e60bbc29a8c3a88ad97896c7c4675cee" exitCode=0
Jul 18 17:18:13.478694 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:13.478625    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq" event={"ID":"95b4ef95-fc98-4eb0-994e-9ba27b9545e6","Type":"ContainerDied","Data":"e8515fc0d7b240f41bdc5463c0567fd2e60bbc29a8c3a88ad97896c7c4675cee"}
Jul 18 17:18:13.479043 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:13.479028    2554 scope.go:117] "RemoveContainer" containerID="e8515fc0d7b240f41bdc5463c0567fd2e60bbc29a8c3a88ad97896c7c4675cee"
Jul 18 17:18:13.480140 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:13.480112    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:18:13.480212 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:13.480197    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:18:13.480258 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:13.480247    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:18:14.462429 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:14.462399    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"
Jul 18 17:18:14.492078 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:14.492039    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq" event={"ID":"95b4ef95-fc98-4eb0-994e-9ba27b9545e6","Type":"ContainerStarted","Data":"4a47427a968d534df8eb931e0cbde82ea878ac34d62773112ee2d1a6f122793d"}
Jul 18 17:18:14.671183 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:14.671135    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:18:14.671934 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:14.671561    2554 patch_prober.go:28] interesting pod/e2e-api-a-73d394-openshift-default-568d759479-kqhjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 18 17:18:14.671934 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:14.671636    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq" podUID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 18 17:18:15.671331 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:15.671291    2554 patch_prober.go:28] interesting pod/e2e-api-a-73d394-openshift-default-568d759479-kqhjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 18 17:18:15.672482 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:15.672053    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq" podUID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 18 17:18:16.671414 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:16.671372    2554 patch_prober.go:28] interesting pod/e2e-api-a-73d394-openshift-default-568d759479-kqhjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 18 17:18:16.672053 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:16.672021    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq" podUID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 18 17:18:17.671375 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:17.671327    2554 patch_prober.go:28] interesting pod/e2e-api-a-73d394-openshift-default-568d759479-kqhjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 18 17:18:17.671590 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:17.671419    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq" podUID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 18 17:18:18.675105 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:18.675072    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:18:18.675488 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:18.675346    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:18:18.676186 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:18.676169    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:18:20.132585 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:20.132555    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:20.301897 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:20.301869    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:28.742100 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:28.742046    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:29.764695 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:29.764655    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:30.073251 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:30.073139    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:30.174374 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:30.174315    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:35.312676 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.312633    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"]
Jul 18 17:18:35.313146 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.312909    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86" podUID="3aa57603-a622-4661-8089-506f303de554" containerName="maas-api" containerID="cri-o://5d3e091da192b8bc8b31c32fe3a4ffd08a294d142a3ebdb43c49f14b0cbc7a29" gracePeriod=30
Jul 18 17:18:35.335931 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.335906    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:35.557002 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.556983    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"
Jul 18 17:18:35.589864 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.589818    2554 generic.go:356] "Generic (PLEG): container finished" podID="3aa57603-a622-4661-8089-506f303de554" containerID="5d3e091da192b8bc8b31c32fe3a4ffd08a294d142a3ebdb43c49f14b0cbc7a29" exitCode=0
Jul 18 17:18:35.590040 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.589882    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"
Jul 18 17:18:35.590040 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.589875    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86" event={"ID":"3aa57603-a622-4661-8089-506f303de554","Type":"ContainerDied","Data":"5d3e091da192b8bc8b31c32fe3a4ffd08a294d142a3ebdb43c49f14b0cbc7a29"}
Jul 18 17:18:35.590040 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.589999    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86" event={"ID":"3aa57603-a622-4661-8089-506f303de554","Type":"ContainerDied","Data":"aeb49e95f711618bd82766253ff61956dbf59e629dfc662f8bee3eaf6a55d422"}
Jul 18 17:18:35.590040 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.590028    2554 scope.go:117] "RemoveContainer" containerID="5d3e091da192b8bc8b31c32fe3a4ffd08a294d142a3ebdb43c49f14b0cbc7a29"
Jul 18 17:18:35.598148 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.598133    2554 scope.go:117] "RemoveContainer" containerID="5d3e091da192b8bc8b31c32fe3a4ffd08a294d142a3ebdb43c49f14b0cbc7a29"
Jul 18 17:18:35.598380 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:18:35.598363    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3e091da192b8bc8b31c32fe3a4ffd08a294d142a3ebdb43c49f14b0cbc7a29\": container with ID starting with 5d3e091da192b8bc8b31c32fe3a4ffd08a294d142a3ebdb43c49f14b0cbc7a29 not found: ID does not exist" containerID="5d3e091da192b8bc8b31c32fe3a4ffd08a294d142a3ebdb43c49f14b0cbc7a29"
Jul 18 17:18:35.598454 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.598393    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3e091da192b8bc8b31c32fe3a4ffd08a294d142a3ebdb43c49f14b0cbc7a29"} err="failed to get container status \"5d3e091da192b8bc8b31c32fe3a4ffd08a294d142a3ebdb43c49f14b0cbc7a29\": rpc error: code = NotFound desc = could not find container \"5d3e091da192b8bc8b31c32fe3a4ffd08a294d142a3ebdb43c49f14b0cbc7a29\": container with ID starting with 5d3e091da192b8bc8b31c32fe3a4ffd08a294d142a3ebdb43c49f14b0cbc7a29 not found: ID does not exist"
Jul 18 17:18:35.690333 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.690297    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa57603-a622-4661-8089-506f303de554-openshift-service-ca\") pod \"3aa57603-a622-4661-8089-506f303de554\" (UID: \"3aa57603-a622-4661-8089-506f303de554\") "
Jul 18 17:18:35.690506 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.690342    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99756\" (UniqueName: \"kubernetes.io/projected/3aa57603-a622-4661-8089-506f303de554-kube-api-access-99756\") pod \"3aa57603-a622-4661-8089-506f303de554\" (UID: \"3aa57603-a622-4661-8089-506f303de554\") "
Jul 18 17:18:35.690506 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.690434    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aa57603-a622-4661-8089-506f303de554-maas-api-tls\") pod \"3aa57603-a622-4661-8089-506f303de554\" (UID: \"3aa57603-a622-4661-8089-506f303de554\") "
Jul 18 17:18:35.690691 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.690666    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa57603-a622-4661-8089-506f303de554-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3aa57603-a622-4661-8089-506f303de554" (UID: "3aa57603-a622-4661-8089-506f303de554"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:18:35.692596 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.692574    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa57603-a622-4661-8089-506f303de554-kube-api-access-99756" (OuterVolumeSpecName: "kube-api-access-99756") pod "3aa57603-a622-4661-8089-506f303de554" (UID: "3aa57603-a622-4661-8089-506f303de554"). InnerVolumeSpecName "kube-api-access-99756". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:18:35.692694 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.692669    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa57603-a622-4661-8089-506f303de554-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3aa57603-a622-4661-8089-506f303de554" (UID: "3aa57603-a622-4661-8089-506f303de554"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:18:35.792080 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.792038    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aa57603-a622-4661-8089-506f303de554-maas-api-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:18:35.792080 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.792074    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa57603-a622-4661-8089-506f303de554-openshift-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:18:35.792080 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.792087    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99756\" (UniqueName: \"kubernetes.io/projected/3aa57603-a622-4661-8089-506f303de554-kube-api-access-99756\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:18:35.906747 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.906662    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"]
Jul 18 17:18:35.910401 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:35.910376    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-73d394-5898c4449b-c5c86"]
Jul 18 17:18:37.766953 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:37.766920    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aa57603-a622-4661-8089-506f303de554" path="/var/lib/kubelet/pods/3aa57603-a622-4661-8089-506f303de554/volumes"
Jul 18 17:18:39.637235 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:39.637201    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:40.025520 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:40.025409    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:45.699049 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:45.699010    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:45.755287 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:45.755255    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:45.937591 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:45.937549    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"]
Jul 18 17:18:45.937887 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:45.937777    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq" podUID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" containerName="istio-proxy" containerID="cri-o://4a47427a968d534df8eb931e0cbde82ea878ac34d62773112ee2d1a6f122793d" gracePeriod=30
Jul 18 17:18:45.957378 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:45.957296    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:46.268592 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:46.268508    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:46.342135 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:46.342106    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-376c15-4nnm2"]
Jul 18 17:18:46.342432 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:46.342420    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa57603-a622-4661-8089-506f303de554" containerName="maas-api"
Jul 18 17:18:46.342474 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:46.342433    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa57603-a622-4661-8089-506f303de554" containerName="maas-api"
Jul 18 17:18:46.342508 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:46.342497    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa57603-a622-4661-8089-506f303de554" containerName="maas-api"
Jul 18 17:18:46.346816 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:46.346802    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-376c15-4nnm2"
Jul 18 17:18:46.350050 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:46.350030    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7rq8\""
Jul 18 17:18:46.356185 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:46.356161    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-376c15-4nnm2"]
Jul 18 17:18:46.487755 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:46.487719    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/792aba95-403d-4594-b385-72cac47a046a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-376c15-4nnm2\" (UID: \"792aba95-403d-4594-b385-72cac47a046a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-376c15-4nnm2"
Jul 18 17:18:46.589225 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:46.589193    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/792aba95-403d-4594-b385-72cac47a046a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-376c15-4nnm2\" (UID: \"792aba95-403d-4594-b385-72cac47a046a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-376c15-4nnm2"
Jul 18 17:18:46.589896 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:46.589876    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/792aba95-403d-4594-b385-72cac47a046a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-376c15-4nnm2\" (UID: \"792aba95-403d-4594-b385-72cac47a046a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-376c15-4nnm2"
Jul 18 17:18:46.656929 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:46.656898    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-376c15-4nnm2"
Jul 18 17:18:46.787878 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:46.787829    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-376c15-4nnm2"]
Jul 18 17:18:46.791458 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:18:46.791422    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792aba95_403d_4594_b385_72cac47a046a.slice/crio-e566bc7792681dc1d849acaf263d8c83542f9128adde0e0f2660734cee236f1e WatchSource:0}: Error finding container e566bc7792681dc1d849acaf263d8c83542f9128adde0e0f2660734cee236f1e: Status 404 returned error can't find the container with id e566bc7792681dc1d849acaf263d8c83542f9128adde0e0f2660734cee236f1e
Jul 18 17:18:46.845417 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:46.845346    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:46.945721 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:46.945685    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:18:47.647315 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:47.647282    2554 generic.go:356] "Generic (PLEG): container finished" podID="792aba95-403d-4594-b385-72cac47a046a" containerID="199cfa1588ffa9000fcf9d001bdd173241c3a429c6d65653db141b9ba7241458" exitCode=0
Jul 18 17:18:47.647503 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:47.647321    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-376c15-4nnm2" event={"ID":"792aba95-403d-4594-b385-72cac47a046a","Type":"ContainerDied","Data":"199cfa1588ffa9000fcf9d001bdd173241c3a429c6d65653db141b9ba7241458"}
Jul 18 17:18:47.647503 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:47.647359    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-376c15-4nnm2" event={"ID":"792aba95-403d-4594-b385-72cac47a046a","Type":"ContainerStarted","Data":"e566bc7792681dc1d849acaf263d8c83542f9128adde0e0f2660734cee236f1e"}
Jul 18 17:18:48.677898 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:48.677869    2554 patch_prober.go:28] interesting pod/e2e-api-a-73d394-openshift-default-568d759479-kqhjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 18 17:18:48.678240 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:48.677921    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq" podUID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 18 17:18:48.775001 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:48.774974    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-376c15-4nnm2"
Jul 18 17:18:48.807965 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:48.807928    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/792aba95-403d-4594-b385-72cac47a046a-maas-api-service-ca\") pod \"792aba95-403d-4594-b385-72cac47a046a\" (UID: \"792aba95-403d-4594-b385-72cac47a046a\") "
Jul 18 17:18:48.808266 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:48.808242    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792aba95-403d-4594-b385-72cac47a046a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "792aba95-403d-4594-b385-72cac47a046a" (UID: "792aba95-403d-4594-b385-72cac47a046a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:18:48.909174 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:48.909086    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/792aba95-403d-4594-b385-72cac47a046a-maas-api-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:18:49.657750 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:49.657722    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-376c15-4nnm2"
Jul 18 17:18:49.657954 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:49.657756    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-376c15-4nnm2" event={"ID":"792aba95-403d-4594-b385-72cac47a046a","Type":"ContainerDied","Data":"e566bc7792681dc1d849acaf263d8c83542f9128adde0e0f2660734cee236f1e"}
Jul 18 17:18:49.657954 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:49.657823    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e566bc7792681dc1d849acaf263d8c83542f9128adde0e0f2660734cee236f1e"
Jul 18 17:18:51.193414 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.193391    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:18:51.228346 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.228283    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-workload-certs\") pod \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") "
Jul 18 17:18:51.228346 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.228316    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istiod-ca-cert\") pod \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") "
Jul 18 17:18:51.228346 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.228337    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-podinfo\") pod \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") "
Jul 18 17:18:51.228610 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.228356    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-workload-socket\") pod \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") "
Jul 18 17:18:51.228610 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.228389    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-data\") pod \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") "
Jul 18 17:18:51.228610 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.228411    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-token\") pod \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") "
Jul 18 17:18:51.228610 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.228467    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-credential-socket\") pod \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") "
Jul 18 17:18:51.228610 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.228493    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx2ps\" (UniqueName: \"kubernetes.io/projected/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-kube-api-access-tx2ps\") pod \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") "
Jul 18 17:18:51.228610 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.228549    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-envoy\") pod \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\" (UID: \"95b4ef95-fc98-4eb0-994e-9ba27b9545e6\") "
Jul 18 17:18:51.228610 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.228598    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "95b4ef95-fc98-4eb0-994e-9ba27b9545e6" (UID: "95b4ef95-fc98-4eb0-994e-9ba27b9545e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:18:51.229000 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.228770    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "95b4ef95-fc98-4eb0-994e-9ba27b9545e6" (UID: "95b4ef95-fc98-4eb0-994e-9ba27b9545e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:18:51.229000 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.228834    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-workload-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:18:51.229000 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.228905    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "95b4ef95-fc98-4eb0-994e-9ba27b9545e6" (UID: "95b4ef95-fc98-4eb0-994e-9ba27b9545e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:18:51.229173 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.229140    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "95b4ef95-fc98-4eb0-994e-9ba27b9545e6" (UID: "95b4ef95-fc98-4eb0-994e-9ba27b9545e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:18:51.229674 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.229653    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-data" (OuterVolumeSpecName: "istio-data") pod "95b4ef95-fc98-4eb0-994e-9ba27b9545e6" (UID: "95b4ef95-fc98-4eb0-994e-9ba27b9545e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:18:51.231150 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.231119    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-kube-api-access-tx2ps" (OuterVolumeSpecName: "kube-api-access-tx2ps") pod "95b4ef95-fc98-4eb0-994e-9ba27b9545e6" (UID: "95b4ef95-fc98-4eb0-994e-9ba27b9545e6"). InnerVolumeSpecName "kube-api-access-tx2ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:18:51.231355 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.231326    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "95b4ef95-fc98-4eb0-994e-9ba27b9545e6" (UID: "95b4ef95-fc98-4eb0-994e-9ba27b9545e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 17:18:51.231465 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.231402    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "95b4ef95-fc98-4eb0-994e-9ba27b9545e6" (UID: "95b4ef95-fc98-4eb0-994e-9ba27b9545e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:18:51.231465 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.231420    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-token" (OuterVolumeSpecName: "istio-token") pod "95b4ef95-fc98-4eb0-994e-9ba27b9545e6" (UID: "95b4ef95-fc98-4eb0-994e-9ba27b9545e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:18:51.330189 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.330149    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-envoy\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:18:51.330189 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.330182    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istiod-ca-cert\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:18:51.330189 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.330191    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-podinfo\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:18:51.330189 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.330199    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-workload-socket\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:18:51.330459 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.330207    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-data\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:18:51.330459 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.330215    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-istio-token\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:18:51.330459 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.330222    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-credential-socket\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:18:51.330459 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.330231    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx2ps\" (UniqueName: \"kubernetes.io/projected/95b4ef95-fc98-4eb0-994e-9ba27b9545e6-kube-api-access-tx2ps\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:18:51.668468 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.668432    2554 generic.go:356] "Generic (PLEG): container finished" podID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" containerID="4a47427a968d534df8eb931e0cbde82ea878ac34d62773112ee2d1a6f122793d" exitCode=0
Jul 18 17:18:51.668667 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.668484    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"
Jul 18 17:18:51.668667 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.668516    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq" event={"ID":"95b4ef95-fc98-4eb0-994e-9ba27b9545e6","Type":"ContainerDied","Data":"4a47427a968d534df8eb931e0cbde82ea878ac34d62773112ee2d1a6f122793d"}
Jul 18 17:18:51.668667 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.668572    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq" event={"ID":"95b4ef95-fc98-4eb0-994e-9ba27b9545e6","Type":"ContainerDied","Data":"f461d5dbc1b22feb9cdb06be7e1173971457d11f48f0934be4906e34891c2fb4"}
Jul 18 17:18:51.668667 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.668595    2554 scope.go:117] "RemoveContainer" containerID="4a47427a968d534df8eb931e0cbde82ea878ac34d62773112ee2d1a6f122793d"
Jul 18 17:18:51.677619 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.677598    2554 scope.go:117] "RemoveContainer" containerID="e8515fc0d7b240f41bdc5463c0567fd2e60bbc29a8c3a88ad97896c7c4675cee"
Jul 18 17:18:51.689134 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.689116    2554 scope.go:117] "RemoveContainer" containerID="4a47427a968d534df8eb931e0cbde82ea878ac34d62773112ee2d1a6f122793d"
Jul 18 17:18:51.689498 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:18:51.689438    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a47427a968d534df8eb931e0cbde82ea878ac34d62773112ee2d1a6f122793d\": container with ID starting with 4a47427a968d534df8eb931e0cbde82ea878ac34d62773112ee2d1a6f122793d not found: ID does not exist" containerID="4a47427a968d534df8eb931e0cbde82ea878ac34d62773112ee2d1a6f122793d"
Jul 18 17:18:51.689602 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.689556    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a47427a968d534df8eb931e0cbde82ea878ac34d62773112ee2d1a6f122793d"} err="failed to get container status \"4a47427a968d534df8eb931e0cbde82ea878ac34d62773112ee2d1a6f122793d\": rpc error: code = NotFound desc = could not find container \"4a47427a968d534df8eb931e0cbde82ea878ac34d62773112ee2d1a6f122793d\": container with ID starting with 4a47427a968d534df8eb931e0cbde82ea878ac34d62773112ee2d1a6f122793d not found: ID does not exist"
Jul 18 17:18:51.689602 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.689582    2554 scope.go:117] "RemoveContainer" containerID="e8515fc0d7b240f41bdc5463c0567fd2e60bbc29a8c3a88ad97896c7c4675cee"
Jul 18 17:18:51.689968 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:18:51.689918    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8515fc0d7b240f41bdc5463c0567fd2e60bbc29a8c3a88ad97896c7c4675cee\": container with ID starting with e8515fc0d7b240f41bdc5463c0567fd2e60bbc29a8c3a88ad97896c7c4675cee not found: ID does not exist" containerID="e8515fc0d7b240f41bdc5463c0567fd2e60bbc29a8c3a88ad97896c7c4675cee"
Jul 18 17:18:51.689968 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.689953    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8515fc0d7b240f41bdc5463c0567fd2e60bbc29a8c3a88ad97896c7c4675cee"} err="failed to get container status \"e8515fc0d7b240f41bdc5463c0567fd2e60bbc29a8c3a88ad97896c7c4675cee\": rpc error: code = NotFound desc = could not find container \"e8515fc0d7b240f41bdc5463c0567fd2e60bbc29a8c3a88ad97896c7c4675cee\": container with ID starting with e8515fc0d7b240f41bdc5463c0567fd2e60bbc29a8c3a88ad97896c7c4675cee not found: ID does not exist"
Jul 18 17:18:51.691389 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.691371    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"]
Jul 18 17:18:51.696631 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.696611    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-73d394-openshift-default-568d759479-kqhjq"]
Jul 18 17:18:51.765946 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:18:51.765913    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" path="/var/lib/kubelet/pods/95b4ef95-fc98-4eb0-994e-9ba27b9545e6/volumes"
Jul 18 17:19:01.854527 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:01.854486    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"]
Jul 18 17:19:01.855045 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:01.854812    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb" podUID="ea291f58-b23d-4d55-88de-d8d4682ba848" containerName="maas-api" containerID="cri-o://aedb16102bfe356d76c1af8af9f83b247f449e4d0022529aaf1358e65cb83867" gracePeriod=30
Jul 18 17:19:01.881814 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:01.881789    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:02.098532 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.098506    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"
Jul 18 17:19:02.228180 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.228096    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxvb6\" (UniqueName: \"kubernetes.io/projected/ea291f58-b23d-4d55-88de-d8d4682ba848-kube-api-access-cxvb6\") pod \"ea291f58-b23d-4d55-88de-d8d4682ba848\" (UID: \"ea291f58-b23d-4d55-88de-d8d4682ba848\") "
Jul 18 17:19:02.228180 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.228142    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea291f58-b23d-4d55-88de-d8d4682ba848-maas-api-tls\") pod \"ea291f58-b23d-4d55-88de-d8d4682ba848\" (UID: \"ea291f58-b23d-4d55-88de-d8d4682ba848\") "
Jul 18 17:19:02.228180 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.228164    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea291f58-b23d-4d55-88de-d8d4682ba848-openshift-service-ca\") pod \"ea291f58-b23d-4d55-88de-d8d4682ba848\" (UID: \"ea291f58-b23d-4d55-88de-d8d4682ba848\") "
Jul 18 17:19:02.228591 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.228566    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea291f58-b23d-4d55-88de-d8d4682ba848-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea291f58-b23d-4d55-88de-d8d4682ba848" (UID: "ea291f58-b23d-4d55-88de-d8d4682ba848"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:19:02.230395 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.230373    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea291f58-b23d-4d55-88de-d8d4682ba848-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea291f58-b23d-4d55-88de-d8d4682ba848" (UID: "ea291f58-b23d-4d55-88de-d8d4682ba848"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:19:02.230513 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.230491    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea291f58-b23d-4d55-88de-d8d4682ba848-kube-api-access-cxvb6" (OuterVolumeSpecName: "kube-api-access-cxvb6") pod "ea291f58-b23d-4d55-88de-d8d4682ba848" (UID: "ea291f58-b23d-4d55-88de-d8d4682ba848"). InnerVolumeSpecName "kube-api-access-cxvb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:19:02.329680 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.329645    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxvb6\" (UniqueName: \"kubernetes.io/projected/ea291f58-b23d-4d55-88de-d8d4682ba848-kube-api-access-cxvb6\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:19:02.329680 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.329674    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea291f58-b23d-4d55-88de-d8d4682ba848-maas-api-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:19:02.329680 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.329684    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea291f58-b23d-4d55-88de-d8d4682ba848-openshift-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:19:02.719708 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.719672    2554 generic.go:356] "Generic (PLEG): container finished" podID="ea291f58-b23d-4d55-88de-d8d4682ba848" containerID="aedb16102bfe356d76c1af8af9f83b247f449e4d0022529aaf1358e65cb83867" exitCode=0
Jul 18 17:19:02.719958 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.719712    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"
Jul 18 17:19:02.719958 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.719755    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb" event={"ID":"ea291f58-b23d-4d55-88de-d8d4682ba848","Type":"ContainerDied","Data":"aedb16102bfe356d76c1af8af9f83b247f449e4d0022529aaf1358e65cb83867"}
Jul 18 17:19:02.719958 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.719802    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb" event={"ID":"ea291f58-b23d-4d55-88de-d8d4682ba848","Type":"ContainerDied","Data":"4f06f53ab3fd40feec88ebbf82d993a755d0710d668c47e030ddfddd08ce2ecb"}
Jul 18 17:19:02.719958 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.719823    2554 scope.go:117] "RemoveContainer" containerID="aedb16102bfe356d76c1af8af9f83b247f449e4d0022529aaf1358e65cb83867"
Jul 18 17:19:02.728562 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.728545    2554 scope.go:117] "RemoveContainer" containerID="aedb16102bfe356d76c1af8af9f83b247f449e4d0022529aaf1358e65cb83867"
Jul 18 17:19:02.728821 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:19:02.728804    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aedb16102bfe356d76c1af8af9f83b247f449e4d0022529aaf1358e65cb83867\": container with ID starting with aedb16102bfe356d76c1af8af9f83b247f449e4d0022529aaf1358e65cb83867 not found: ID does not exist" containerID="aedb16102bfe356d76c1af8af9f83b247f449e4d0022529aaf1358e65cb83867"
Jul 18 17:19:02.728914 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.728828    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aedb16102bfe356d76c1af8af9f83b247f449e4d0022529aaf1358e65cb83867"} err="failed to get container status \"aedb16102bfe356d76c1af8af9f83b247f449e4d0022529aaf1358e65cb83867\": rpc error: code = NotFound desc = could not find container \"aedb16102bfe356d76c1af8af9f83b247f449e4d0022529aaf1358e65cb83867\": container with ID starting with aedb16102bfe356d76c1af8af9f83b247f449e4d0022529aaf1358e65cb83867 not found: ID does not exist"
Jul 18 17:19:02.740206 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.740180    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"]
Jul 18 17:19:02.743686 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:02.743662    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-376c15-7dbc488fc4-4fwdb"]
Jul 18 17:19:03.765608 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:03.765566    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea291f58-b23d-4d55-88de-d8d4682ba848" path="/var/lib/kubelet/pods/ea291f58-b23d-4d55-88de-d8d4682ba848/volumes"
Jul 18 17:19:06.413188 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:06.413153    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:06.417410 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:06.417384    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-376c15-4nnm2"]
Jul 18 17:19:06.419973 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:06.419952    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-376c15-4nnm2"]
Jul 18 17:19:06.783476 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:06.783385    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:07.768001 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:07.767963    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="792aba95-403d-4594-b385-72cac47a046a" path="/var/lib/kubelet/pods/792aba95-403d-4594-b385-72cac47a046a/volumes"
Jul 18 17:19:12.550582 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:12.550552    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:12.648395 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:12.648358    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:12.802324 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:12.802234    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:13.061595 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.061511    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:13.370631 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.370602    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:13.654781 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.654699    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"]
Jul 18 17:19:13.655242 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.655191    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" containerName="istio-proxy"
Jul 18 17:19:13.655242 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.655208    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" containerName="istio-proxy"
Jul 18 17:19:13.655242 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.655238    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792aba95-403d-4594-b385-72cac47a046a" containerName="revoke-keys"
Jul 18 17:19:13.655401 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.655247    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="792aba95-403d-4594-b385-72cac47a046a" containerName="revoke-keys"
Jul 18 17:19:13.655401 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.655258    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea291f58-b23d-4d55-88de-d8d4682ba848" containerName="maas-api"
Jul 18 17:19:13.655401 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.655266    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea291f58-b23d-4d55-88de-d8d4682ba848" containerName="maas-api"
Jul 18 17:19:13.655401 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.655359    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" containerName="istio-proxy"
Jul 18 17:19:13.655401 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.655370    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" containerName="istio-proxy"
Jul 18 17:19:13.655401 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.655383    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="792aba95-403d-4594-b385-72cac47a046a" containerName="revoke-keys"
Jul 18 17:19:13.655401 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.655392    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea291f58-b23d-4d55-88de-d8d4682ba848" containerName="maas-api"
Jul 18 17:19:13.659706 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.659686    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.662397 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.662375    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 18 17:19:13.662566 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.662549    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-78538d-openshift-default-dockercfg-mrq4x\""
Jul 18 17:19:13.670386 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.670363    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"]
Jul 18 17:19:13.681539 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.681510    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:13.727397 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.727356    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-credential-socket\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.727545 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.727415    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afec0482-b062-4758-b518-850636637cf1-istio-podinfo\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.727545 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.727488    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afec0482-b062-4758-b518-850636637cf1-istio-token\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.727545 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.727536    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-workload-certs\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.727664 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.727571    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afec0482-b062-4758-b518-850636637cf1-istiod-ca-cert\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.727664 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.727598    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-istio-data\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.727664 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.727615    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-workload-socket\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.727757 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.727678    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-istio-envoy\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.727757 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.727701    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4mvd\" (UniqueName: \"kubernetes.io/projected/afec0482-b062-4758-b518-850636637cf1-kube-api-access-h4mvd\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.828696 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.828659    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-istio-data\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.828696 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.828697    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-workload-socket\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.828985 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.828722    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-istio-envoy\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.828985 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.828766    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4mvd\" (UniqueName: \"kubernetes.io/projected/afec0482-b062-4758-b518-850636637cf1-kube-api-access-h4mvd\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.829107 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.829004    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-credential-socket\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.829107 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.829098    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afec0482-b062-4758-b518-850636637cf1-istio-podinfo\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.829221 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.829141    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afec0482-b062-4758-b518-850636637cf1-istio-token\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.829221 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.829150    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-workload-socket\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.829221 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.829210    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-workload-certs\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.829381 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.829223    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-istio-data\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.829381 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.829248    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afec0482-b062-4758-b518-850636637cf1-istiod-ca-cert\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.829381 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.829326    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-credential-socket\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.829543 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.829499    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-workload-certs\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.829897 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.829805    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afec0482-b062-4758-b518-850636637cf1-istiod-ca-cert\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.831594 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.831570    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-istio-envoy\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.831793 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.831778    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afec0482-b062-4758-b518-850636637cf1-istio-podinfo\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.836992 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.836974    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afec0482-b062-4758-b518-850636637cf1-istio-token\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.837235 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.837218    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4mvd\" (UniqueName: \"kubernetes.io/projected/afec0482-b062-4758-b518-850636637cf1-kube-api-access-h4mvd\") pod \"e2e-shared-a-78538d-openshift-default-8c795f47-6jg45\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") " pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:13.970172 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:13.970069    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:14.021962 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:14.021626    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:14.108656 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:14.108597    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"]
Jul 18 17:19:14.111411 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:19:14.111386    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafec0482_b062_4758_b518_850636637cf1.slice/crio-7ebc2b5c06f6fa14f82e15f81b5aba8707c7d4309360b8d68b108bdc9bfca749 WatchSource:0}: Error finding container 7ebc2b5c06f6fa14f82e15f81b5aba8707c7d4309360b8d68b108bdc9bfca749: Status 404 returned error can't find the container with id 7ebc2b5c06f6fa14f82e15f81b5aba8707c7d4309360b8d68b108bdc9bfca749
Jul 18 17:19:14.113446 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:14.113415    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:19:14.113551 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:14.113491    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:19:14.113621 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:14.113550    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:19:14.775230 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:14.775192    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45" event={"ID":"afec0482-b062-4758-b518-850636637cf1","Type":"ContainerStarted","Data":"63125580a827dc2da7ba439b321cdc13bfb9104af2d3d9dce39b32926edf9e79"}
Jul 18 17:19:14.775646 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:14.775282    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45" event={"ID":"afec0482-b062-4758-b518-850636637cf1","Type":"ContainerStarted","Data":"7ebc2b5c06f6fa14f82e15f81b5aba8707c7d4309360b8d68b108bdc9bfca749"}
Jul 18 17:19:14.797460 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:14.797413    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45" podStartSLOduration=1.797399065 podStartE2EDuration="1.797399065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:19:14.795387296 +0000 UTC m=+2573.493217981" watchObservedRunningTime="2026-07-18 17:19:14.797399065 +0000 UTC m=+2573.495229801"
Jul 18 17:19:14.970222 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:14.970188    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:15.975510 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:15.975482    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:16.737904 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.737833    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:16.784864 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.784820    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:16.785652 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.785634    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:16.823841 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.823814    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:16.853542 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.853516    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"]
Jul 18 17:19:16.853903 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.853888    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" containerName="istio-proxy"
Jul 18 17:19:16.853976 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.853904    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="95b4ef95-fc98-4eb0-994e-9ba27b9545e6" containerName="istio-proxy"
Jul 18 17:19:16.856417 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.856400    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"
Jul 18 17:19:16.858902 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.858874    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 18 17:19:16.859126 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.859105    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-78538d-serving-cert\""
Jul 18 17:19:16.859244 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.859163    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 18 17:19:16.859244 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.859216    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2pqsv\""
Jul 18 17:19:16.867016 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.866990    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"]
Jul 18 17:19:16.923924 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.923897    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:16.960675 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.960646    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56\" (UID: \"3fe8387d-8eed-47a7-a939-75ab5c6aecb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"
Jul 18 17:19:16.960813 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.960715    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-maas-api-tls\") pod \"maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56\" (UID: \"3fe8387d-8eed-47a7-a939-75ab5c6aecb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"
Jul 18 17:19:16.960895 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:16.960808    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vkzj\" (UniqueName: \"kubernetes.io/projected/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-kube-api-access-4vkzj\") pod \"maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56\" (UID: \"3fe8387d-8eed-47a7-a939-75ab5c6aecb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"
Jul 18 17:19:17.061961 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.061837    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-maas-api-tls\") pod \"maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56\" (UID: \"3fe8387d-8eed-47a7-a939-75ab5c6aecb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"
Jul 18 17:19:17.062415 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.062135    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vkzj\" (UniqueName: \"kubernetes.io/projected/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-kube-api-access-4vkzj\") pod \"maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56\" (UID: \"3fe8387d-8eed-47a7-a939-75ab5c6aecb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"
Jul 18 17:19:17.062415 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.062185    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56\" (UID: \"3fe8387d-8eed-47a7-a939-75ab5c6aecb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"
Jul 18 17:19:17.062872 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.062819    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56\" (UID: \"3fe8387d-8eed-47a7-a939-75ab5c6aecb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"
Jul 18 17:19:17.064472 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.064454    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-maas-api-tls\") pod \"maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56\" (UID: \"3fe8387d-8eed-47a7-a939-75ab5c6aecb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"
Jul 18 17:19:17.070190 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.070167    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vkzj\" (UniqueName: \"kubernetes.io/projected/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-kube-api-access-4vkzj\") pod \"maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56\" (UID: \"3fe8387d-8eed-47a7-a939-75ab5c6aecb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"
Jul 18 17:19:17.111222 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.111194    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:17.171022 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.170988    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"
Jul 18 17:19:17.306185 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.306117    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"]
Jul 18 17:19:17.308208 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:19:17.308183    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe8387d_8eed_47a7_a939_75ab5c6aecb6.slice/crio-4451547b8840bd9d76238b7e46ee2b0fe5b35e145f1408273a945649748fefaf WatchSource:0}: Error finding container 4451547b8840bd9d76238b7e46ee2b0fe5b35e145f1408273a945649748fefaf: Status 404 returned error can't find the container with id 4451547b8840bd9d76238b7e46ee2b0fe5b35e145f1408273a945649748fefaf
Jul 18 17:19:17.309770 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.309752    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 17:19:17.405320 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.405284    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:17.508158 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.508117    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:17.790526 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.790494    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56" event={"ID":"3fe8387d-8eed-47a7-a939-75ab5c6aecb6","Type":"ContainerStarted","Data":"34955f19c58939c5aa9c1ec75b133c8f538df9380a2359f1912d26cd65f64482"}
Jul 18 17:19:17.790806 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.790788    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"
Jul 18 17:19:17.790926 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.790829    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56" event={"ID":"3fe8387d-8eed-47a7-a939-75ab5c6aecb6","Type":"ContainerStarted","Data":"4451547b8840bd9d76238b7e46ee2b0fe5b35e145f1408273a945649748fefaf"}
Jul 18 17:19:17.813246 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:17.813204    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56" podStartSLOduration=1.432025175 podStartE2EDuration="1.813188543s" totalImagesPullingTime="381.163368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:19:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:19:17.811559238 +0000 UTC m=+2576.509389923" watchObservedRunningTime="2026-07-18 17:19:17.813188543 +0000 UTC m=+2576.511019226"
Jul 18 17:19:21.211611 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:21.211570    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:21.552320 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:21.552225    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:21.869278 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:21.869248    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:22.062316 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:22.062288    2554 scope.go:117] "RemoveContainer" containerID="28ab753e693dbd22ef3167f13661fbd522c6f93434e6e8243c2e0560514a1dbe"
Jul 18 17:19:22.452561 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:22.452522    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:24.599003 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.598970    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:24.675347 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.675311    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"]
Jul 18 17:19:24.680717 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.680697    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"
Jul 18 17:19:24.683064 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.683044    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-497f21-serving-cert\""
Jul 18 17:19:24.685644 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.685617    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:24.687897 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.687873    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"]
Jul 18 17:19:24.747021 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.746981    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:24.806615 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.806576    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"
Jul 18 17:19:24.832583 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.832551    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a38bdb90-a757-4169-8864-d08232ce0efe-maas-api-tls\") pod \"maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p\" (UID: \"a38bdb90-a757-4169-8864-d08232ce0efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"
Jul 18 17:19:24.833043 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.833015    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38bdb90-a757-4169-8864-d08232ce0efe-openshift-service-ca\") pod \"maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p\" (UID: \"a38bdb90-a757-4169-8864-d08232ce0efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"
Jul 18 17:19:24.833129 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.833087    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn2dc\" (UniqueName: \"kubernetes.io/projected/a38bdb90-a757-4169-8864-d08232ce0efe-kube-api-access-tn2dc\") pod \"maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p\" (UID: \"a38bdb90-a757-4169-8864-d08232ce0efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"
Jul 18 17:19:24.934112 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.934032    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn2dc\" (UniqueName: \"kubernetes.io/projected/a38bdb90-a757-4169-8864-d08232ce0efe-kube-api-access-tn2dc\") pod \"maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p\" (UID: \"a38bdb90-a757-4169-8864-d08232ce0efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"
Jul 18 17:19:24.934402 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.934365    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a38bdb90-a757-4169-8864-d08232ce0efe-maas-api-tls\") pod \"maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p\" (UID: \"a38bdb90-a757-4169-8864-d08232ce0efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"
Jul 18 17:19:24.934654 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.934627    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38bdb90-a757-4169-8864-d08232ce0efe-openshift-service-ca\") pod \"maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p\" (UID: \"a38bdb90-a757-4169-8864-d08232ce0efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"
Jul 18 17:19:24.935271 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.935252    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38bdb90-a757-4169-8864-d08232ce0efe-openshift-service-ca\") pod \"maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p\" (UID: \"a38bdb90-a757-4169-8864-d08232ce0efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"
Jul 18 17:19:24.937064 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.937045    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a38bdb90-a757-4169-8864-d08232ce0efe-maas-api-tls\") pod \"maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p\" (UID: \"a38bdb90-a757-4169-8864-d08232ce0efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"
Jul 18 17:19:24.942829 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.942804    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn2dc\" (UniqueName: \"kubernetes.io/projected/a38bdb90-a757-4169-8864-d08232ce0efe-kube-api-access-tn2dc\") pod \"maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p\" (UID: \"a38bdb90-a757-4169-8864-d08232ce0efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"
Jul 18 17:19:24.949786 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.949752    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:24.991971 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:24.991946    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"
Jul 18 17:19:25.127450 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:25.127421    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"]
Jul 18 17:19:25.129442 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:19:25.129409    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38bdb90_a757_4169_8864_d08232ce0efe.slice/crio-9c763fbfdb127d848f147b6b84ba3a6dd3e3c51884d8ce6ad478c6d4cc71928c WatchSource:0}: Error finding container 9c763fbfdb127d848f147b6b84ba3a6dd3e3c51884d8ce6ad478c6d4cc71928c: Status 404 returned error can't find the container with id 9c763fbfdb127d848f147b6b84ba3a6dd3e3c51884d8ce6ad478c6d4cc71928c
Jul 18 17:19:25.347506 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:25.347476    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:25.448880 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:25.448827    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:25.828435 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:25.828403    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p" event={"ID":"a38bdb90-a757-4169-8864-d08232ce0efe","Type":"ContainerStarted","Data":"f6b8d3800bae15bc21f87b68652c4d666bd1dd2499dda43110719b78818ebc06"}
Jul 18 17:19:25.828861 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:25.828462    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p" event={"ID":"a38bdb90-a757-4169-8864-d08232ce0efe","Type":"ContainerStarted","Data":"9c763fbfdb127d848f147b6b84ba3a6dd3e3c51884d8ce6ad478c6d4cc71928c"}
Jul 18 17:19:25.828861 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:25.828504    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"
Jul 18 17:19:25.846753 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:25.846709    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p" podStartSLOduration=1.5188985050000001 podStartE2EDuration="1.846694268s" totalImagesPullingTime="327.795763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:19:25.84645008 +0000 UTC m=+2584.544280765" watchObservedRunningTime="2026-07-18 17:19:25.846694268 +0000 UTC m=+2584.544524951"
Jul 18 17:19:29.082200 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:29.082145    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:31.837930 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:31.837903    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"
Jul 18 17:19:32.814917 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:32.814882    2554 patch_prober.go:28] interesting pod/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 18 17:19:32.815112 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:32.814972    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45" podUID="afec0482-b062-4758-b518-850636637cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 18 17:19:32.863857 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:32.863805    2554 generic.go:356] "Generic (PLEG): container finished" podID="afec0482-b062-4758-b518-850636637cf1" containerID="63125580a827dc2da7ba439b321cdc13bfb9104af2d3d9dce39b32926edf9e79" exitCode=0
Jul 18 17:19:32.864216 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:32.863883    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45" event={"ID":"afec0482-b062-4758-b518-850636637cf1","Type":"ContainerDied","Data":"63125580a827dc2da7ba439b321cdc13bfb9104af2d3d9dce39b32926edf9e79"}
Jul 18 17:19:32.864273 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:32.864260    2554 scope.go:117] "RemoveContainer" containerID="63125580a827dc2da7ba439b321cdc13bfb9104af2d3d9dce39b32926edf9e79"
Jul 18 17:19:32.865260 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:32.865232    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:19:32.865314 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:32.865298    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:19:32.865349 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:32.865331    2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 18 17:19:33.870934 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:33.870901    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45" event={"ID":"afec0482-b062-4758-b518-850636637cf1","Type":"ContainerStarted","Data":"33a0d083d3b503ee33ca5e52f1fc70485e10574eeac4833364ce7cacafd26219"}
Jul 18 17:19:33.970729 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:33.970695    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:33.970938 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:33.970897    2554 patch_prober.go:28] interesting pod/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 18 17:19:33.970994 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:33.970943    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45" podUID="afec0482-b062-4758-b518-850636637cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 18 17:19:34.971671 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:34.971207    2554 patch_prober.go:28] interesting pod/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 18 17:19:34.971671 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:34.971279    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45" podUID="afec0482-b062-4758-b518-850636637cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 18 17:19:35.970968 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:35.970922    2554 patch_prober.go:28] interesting pod/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 18 17:19:35.971183 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:35.970992    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45" podUID="afec0482-b062-4758-b518-850636637cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 18 17:19:36.970694 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:36.970653    2554 patch_prober.go:28] interesting pod/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 18 17:19:36.971305 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:36.971267    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45" podUID="afec0482-b062-4758-b518-850636637cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 18 17:19:37.971524 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:37.971475    2554 patch_prober.go:28] interesting pod/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 18 17:19:37.972049 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:37.971565    2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45" podUID="afec0482-b062-4758-b518-850636637cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 18 17:19:38.264471 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:38.264391    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:38.366986 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:38.366955    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:38.974175 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:38.974148    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:38.974567 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:38.974393    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:38.975036 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:38.975022    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:19:42.383805 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.383772    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"]
Jul 18 17:19:42.386562 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.386541    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.390228 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.390208    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-78538d\"/\"kube-root-ca.crt\""
Jul 18 17:19:42.390326 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.390211    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-78538d\"/\"openshift-service-ca.crt\""
Jul 18 17:19:42.390326 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.390249    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-78538d\"/\"auth-test-model-78538d-kserve-self-signed-certs\""
Jul 18 17:19:42.390326 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.390229    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-78538d\"/\"default-dockercfg-r27rs\""
Jul 18 17:19:42.395284 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.395261    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"]
Jul 18 17:19:42.412363 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.412342    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:42.487798 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.487763    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6w5\" (UniqueName: \"kubernetes.io/projected/9de71573-87fe-46c1-8cd2-53c32fd2e860-kube-api-access-tb6w5\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.488054 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.487814    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-model-cache\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.488054 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.487929    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-tmp-dir\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.488054 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.488008    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-home\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.488212 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.488056    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de71573-87fe-46c1-8cd2-53c32fd2e860-tls-certs\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.488212 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.488100    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-dshm\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.589122 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.589085    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-model-cache\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.589312 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.589140    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-tmp-dir\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.589312 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.589170    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-home\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.589312 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.589194    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de71573-87fe-46c1-8cd2-53c32fd2e860-tls-certs\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.589312 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.589222    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-dshm\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.589312 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.589268    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6w5\" (UniqueName: \"kubernetes.io/projected/9de71573-87fe-46c1-8cd2-53c32fd2e860-kube-api-access-tb6w5\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.589610 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.589513    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-model-cache\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.589610 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.589591    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-tmp-dir\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.589610 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.589603    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-home\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.591637 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.591610    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-dshm\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.591953 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.591938    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de71573-87fe-46c1-8cd2-53c32fd2e860-tls-certs\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.597693 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.597673    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6w5\" (UniqueName: \"kubernetes.io/projected/9de71573-87fe-46c1-8cd2-53c32fd2e860-kube-api-access-tb6w5\") pod \"auth-test-model-78538d-kserve-565db4b57f-27gjp\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") " pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.696363 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.696267    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:42.827254 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.827221    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"]
Jul 18 17:19:42.829959 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:19:42.829932    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9de71573_87fe_46c1_8cd2_53c32fd2e860.slice/crio-c252d04c9380fbe1ce11b53f275b5dadad968c74b894871998a02592b986047b WatchSource:0}: Error finding container c252d04c9380fbe1ce11b53f275b5dadad968c74b894871998a02592b986047b: Status 404 returned error can't find the container with id c252d04c9380fbe1ce11b53f275b5dadad968c74b894871998a02592b986047b
Jul 18 17:19:42.919129 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.919096    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp" event={"ID":"9de71573-87fe-46c1-8cd2-53c32fd2e860","Type":"ContainerStarted","Data":"98b93dabb14963e3e3fca0791d0a35b77340ae838550c41faad259699afdc9f6"}
Jul 18 17:19:42.919309 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.919138    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp" event={"ID":"9de71573-87fe-46c1-8cd2-53c32fd2e860","Type":"ContainerStarted","Data":"c252d04c9380fbe1ce11b53f275b5dadad968c74b894871998a02592b986047b"}
Jul 18 17:19:42.936208 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:42.936161    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp" podStartSLOduration=0.936146112 podStartE2EDuration="936.146112ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:19:42.93473081 +0000 UTC m=+2601.632561494" watchObservedRunningTime="2026-07-18 17:19:42.936146112 +0000 UTC m=+2601.633976796"
Jul 18 17:19:46.194135 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:46.194093    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:46.294390 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:46.294355    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:19:52.696696 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:52.696655    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:52.697211 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:52.696712    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:52.709244 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:52.709221    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:19:52.976308 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:19:52.976227    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:20:09.172315 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.172280    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"]
Jul 18 17:20:09.175707 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.175690    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.178590 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.178561    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-497f21\"/\"auth-test-model-497f21-kserve-self-signed-certs\""
Jul 18 17:20:09.180442 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.180414    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-497f21\"/\"openshift-service-ca.crt\""
Jul 18 17:20:09.180682 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.180657    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-497f21\"/\"kube-root-ca.crt\""
Jul 18 17:20:09.180922 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.180900    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-497f21\"/\"default-dockercfg-dvrp4\""
Jul 18 17:20:09.198123 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.198098    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"]
Jul 18 17:20:09.218242 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.218215    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:20:09.334654 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.334620    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-home\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.334654 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.334657    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8q76\" (UniqueName: \"kubernetes.io/projected/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-kube-api-access-c8q76\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.334906 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.334676    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-tmp-dir\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.334906 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.334736    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-dshm\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.334906 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.334752    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-model-cache\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.334906 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.334818    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-tls-certs\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.435956 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.435866    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-dshm\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.435956 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.435904    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-model-cache\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.436184 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.435977    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-tls-certs\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.436184 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.436046    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-home\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.436184 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.436073    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8q76\" (UniqueName: \"kubernetes.io/projected/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-kube-api-access-c8q76\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.436184 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.436100    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-tmp-dir\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.436410 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.436352    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-model-cache\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.436410 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.436363    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-home\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.436509 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.436443    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-tmp-dir\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.438146 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.438125    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-dshm\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.438554 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.438539    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-tls-certs\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.444318 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.444295    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8q76\" (UniqueName: \"kubernetes.io/projected/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-kube-api-access-c8q76\") pod \"auth-test-model-497f21-kserve-68d55457bc-67r4x\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") " pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.488379 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.488355    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:09.622747 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:09.622721    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"]
Jul 18 17:20:09.624707 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:20:09.624673    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6ac8f08_9369_42a9_8f5d_838f0ba4d21d.slice/crio-b71de1e87cd99214273984ed73c4dd7921977230431a3db9a0dc5e7e10a38db5 WatchSource:0}: Error finding container b71de1e87cd99214273984ed73c4dd7921977230431a3db9a0dc5e7e10a38db5: Status 404 returned error can't find the container with id b71de1e87cd99214273984ed73c4dd7921977230431a3db9a0dc5e7e10a38db5
Jul 18 17:20:10.044495 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:10.044465    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x" event={"ID":"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d","Type":"ContainerStarted","Data":"045840ae16d1320ba2581d951d411c009ee75e2a4c53cffd2438b4048e1addc2"}
Jul 18 17:20:10.044495 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:10.044502    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x" event={"ID":"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d","Type":"ContainerStarted","Data":"b71de1e87cd99214273984ed73c4dd7921977230431a3db9a0dc5e7e10a38db5"}
Jul 18 17:20:10.062530 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:10.062486    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x" podStartSLOduration=1.062471552 podStartE2EDuration="1.062471552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:20:10.061706908 +0000 UTC m=+2628.759537592" watchObservedRunningTime="2026-07-18 17:20:10.062471552 +0000 UTC m=+2628.760302235"
Jul 18 17:20:10.662722 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:10.662690    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:20:10.765180 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:10.765141    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:20:19.489087 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:19.489042    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:19.489512 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:19.489339    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:19.501960 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:19.501938    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:20.102838 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:20.102810    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:20:33.607621 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:20:33.607586    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:21:50.757696 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:21:50.757611    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:21:50.815647 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:21:50.815609    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:22:02.640724 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:02.640691    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:22:44.145534 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:44.145449    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"]
Jul 18 17:22:44.161626 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:44.161596    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:22:45.135960 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:45.135928    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"]
Jul 18 17:22:45.159238 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:45.159209    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:22:48.449012 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:48.448977    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:22:59.166193 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.166130    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp" podUID="9de71573-87fe-46c1-8cd2-53c32fd2e860" containerName="main" containerID="cri-o://98b93dabb14963e3e3fca0791d0a35b77340ae838550c41faad259699afdc9f6" gracePeriod=45
Jul 18 17:22:59.422976 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.422921    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:22:59.436081 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.436055    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-model-cache\") pod \"9de71573-87fe-46c1-8cd2-53c32fd2e860\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") "
Jul 18 17:22:59.436192 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.436113    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-home\") pod \"9de71573-87fe-46c1-8cd2-53c32fd2e860\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") "
Jul 18 17:22:59.436192 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.436143    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de71573-87fe-46c1-8cd2-53c32fd2e860-tls-certs\") pod \"9de71573-87fe-46c1-8cd2-53c32fd2e860\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") "
Jul 18 17:22:59.436192 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.436177    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-tmp-dir\") pod \"9de71573-87fe-46c1-8cd2-53c32fd2e860\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") "
Jul 18 17:22:59.436342 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.436196    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-dshm\") pod \"9de71573-87fe-46c1-8cd2-53c32fd2e860\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") "
Jul 18 17:22:59.436342 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.436225    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb6w5\" (UniqueName: \"kubernetes.io/projected/9de71573-87fe-46c1-8cd2-53c32fd2e860-kube-api-access-tb6w5\") pod \"9de71573-87fe-46c1-8cd2-53c32fd2e860\" (UID: \"9de71573-87fe-46c1-8cd2-53c32fd2e860\") "
Jul 18 17:22:59.436445 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.436319    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-model-cache" (OuterVolumeSpecName: "model-cache") pod "9de71573-87fe-46c1-8cd2-53c32fd2e860" (UID: "9de71573-87fe-46c1-8cd2-53c32fd2e860"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:22:59.436500 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.436447    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9de71573-87fe-46c1-8cd2-53c32fd2e860" (UID: "9de71573-87fe-46c1-8cd2-53c32fd2e860"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:22:59.436500 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.436460    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-home" (OuterVolumeSpecName: "home") pod "9de71573-87fe-46c1-8cd2-53c32fd2e860" (UID: "9de71573-87fe-46c1-8cd2-53c32fd2e860"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:22:59.436600 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.436566    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-model-cache\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:22:59.436600 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.436584    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-home\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:22:59.436677 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.436597    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-tmp-dir\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:22:59.438437 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.438413    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-dshm" (OuterVolumeSpecName: "dshm") pod "9de71573-87fe-46c1-8cd2-53c32fd2e860" (UID: "9de71573-87fe-46c1-8cd2-53c32fd2e860"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:22:59.438543 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.438464    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de71573-87fe-46c1-8cd2-53c32fd2e860-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9de71573-87fe-46c1-8cd2-53c32fd2e860" (UID: "9de71573-87fe-46c1-8cd2-53c32fd2e860"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:22:59.438667 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.438650    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de71573-87fe-46c1-8cd2-53c32fd2e860-kube-api-access-tb6w5" (OuterVolumeSpecName: "kube-api-access-tb6w5") pod "9de71573-87fe-46c1-8cd2-53c32fd2e860" (UID: "9de71573-87fe-46c1-8cd2-53c32fd2e860"). InnerVolumeSpecName "kube-api-access-tb6w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:22:59.537654 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.537621    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9de71573-87fe-46c1-8cd2-53c32fd2e860-tls-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:22:59.537654 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.537646    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9de71573-87fe-46c1-8cd2-53c32fd2e860-dshm\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:22:59.537654 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.537656    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb6w5\" (UniqueName: \"kubernetes.io/projected/9de71573-87fe-46c1-8cd2-53c32fd2e860-kube-api-access-tb6w5\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:22:59.833563 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.833526    2554 generic.go:356] "Generic (PLEG): container finished" podID="9de71573-87fe-46c1-8cd2-53c32fd2e860" containerID="98b93dabb14963e3e3fca0791d0a35b77340ae838550c41faad259699afdc9f6" exitCode=0
Jul 18 17:22:59.833728 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.833573    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"
Jul 18 17:22:59.833728 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.833615    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp" event={"ID":"9de71573-87fe-46c1-8cd2-53c32fd2e860","Type":"ContainerDied","Data":"98b93dabb14963e3e3fca0791d0a35b77340ae838550c41faad259699afdc9f6"}
Jul 18 17:22:59.833728 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.833671    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp" event={"ID":"9de71573-87fe-46c1-8cd2-53c32fd2e860","Type":"ContainerDied","Data":"c252d04c9380fbe1ce11b53f275b5dadad968c74b894871998a02592b986047b"}
Jul 18 17:22:59.833728 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.833694    2554 scope.go:117] "RemoveContainer" containerID="98b93dabb14963e3e3fca0791d0a35b77340ae838550c41faad259699afdc9f6"
Jul 18 17:22:59.842375 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.842347    2554 scope.go:117] "RemoveContainer" containerID="98b93dabb14963e3e3fca0791d0a35b77340ae838550c41faad259699afdc9f6"
Jul 18 17:22:59.842630 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:22:59.842612    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b93dabb14963e3e3fca0791d0a35b77340ae838550c41faad259699afdc9f6\": container with ID starting with 98b93dabb14963e3e3fca0791d0a35b77340ae838550c41faad259699afdc9f6 not found: ID does not exist" containerID="98b93dabb14963e3e3fca0791d0a35b77340ae838550c41faad259699afdc9f6"
Jul 18 17:22:59.842686 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.842639    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b93dabb14963e3e3fca0791d0a35b77340ae838550c41faad259699afdc9f6"} err="failed to get container status \"98b93dabb14963e3e3fca0791d0a35b77340ae838550c41faad259699afdc9f6\": rpc error: code = NotFound desc = could not find container \"98b93dabb14963e3e3fca0791d0a35b77340ae838550c41faad259699afdc9f6\": container with ID starting with 98b93dabb14963e3e3fca0791d0a35b77340ae838550c41faad259699afdc9f6 not found: ID does not exist"
Jul 18 17:22:59.851759 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.851734    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"]
Jul 18 17:22:59.855571 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:22:59.855551    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-78538d/auth-test-model-78538d-kserve-565db4b57f-27gjp"]
Jul 18 17:23:00.157288 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.157249    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x" podUID="f6ac8f08-9369-42a9-8f5d-838f0ba4d21d" containerName="main" containerID="cri-o://045840ae16d1320ba2581d951d411c009ee75e2a4c53cffd2438b4048e1addc2" gracePeriod=45
Jul 18 17:23:00.402940 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.402919    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:23:00.445969 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.445897    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-model-cache\") pod \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") "
Jul 18 17:23:00.445969 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.445968    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-tls-certs\") pod \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") "
Jul 18 17:23:00.446167 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.445986    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-home\") pod \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") "
Jul 18 17:23:00.446167 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.446002    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-tmp-dir\") pod \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") "
Jul 18 17:23:00.446167 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.446032    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8q76\" (UniqueName: \"kubernetes.io/projected/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-kube-api-access-c8q76\") pod \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") "
Jul 18 17:23:00.446167 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.446067    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-dshm\") pod \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\" (UID: \"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d\") "
Jul 18 17:23:00.446386 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.446216    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6ac8f08-9369-42a9-8f5d-838f0ba4d21d" (UID: "f6ac8f08-9369-42a9-8f5d-838f0ba4d21d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:23:00.446386 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.446267    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-home" (OuterVolumeSpecName: "home") pod "f6ac8f08-9369-42a9-8f5d-838f0ba4d21d" (UID: "f6ac8f08-9369-42a9-8f5d-838f0ba4d21d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:23:00.446386 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.446307    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6ac8f08-9369-42a9-8f5d-838f0ba4d21d" (UID: "f6ac8f08-9369-42a9-8f5d-838f0ba4d21d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:23:00.446386 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.446370    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-model-cache\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:23:00.446386 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.446380    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-home\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:23:00.446386 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.446390    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-tmp-dir\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:23:00.448463 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.448440    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-dshm" (OuterVolumeSpecName: "dshm") pod "f6ac8f08-9369-42a9-8f5d-838f0ba4d21d" (UID: "f6ac8f08-9369-42a9-8f5d-838f0ba4d21d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:23:00.448571 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.448528    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6ac8f08-9369-42a9-8f5d-838f0ba4d21d" (UID: "f6ac8f08-9369-42a9-8f5d-838f0ba4d21d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:23:00.448571 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.448559    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-kube-api-access-c8q76" (OuterVolumeSpecName: "kube-api-access-c8q76") pod "f6ac8f08-9369-42a9-8f5d-838f0ba4d21d" (UID: "f6ac8f08-9369-42a9-8f5d-838f0ba4d21d"). InnerVolumeSpecName "kube-api-access-c8q76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:23:00.546775 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.546741    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-tls-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:23:00.546775 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.546769    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8q76\" (UniqueName: \"kubernetes.io/projected/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-kube-api-access-c8q76\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:23:00.546775 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.546779    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d-dshm\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:23:00.839741 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.839712    2554 generic.go:356] "Generic (PLEG): container finished" podID="f6ac8f08-9369-42a9-8f5d-838f0ba4d21d" containerID="045840ae16d1320ba2581d951d411c009ee75e2a4c53cffd2438b4048e1addc2" exitCode=0
Jul 18 17:23:00.839971 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.839757    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"
Jul 18 17:23:00.839971 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.839796    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x" event={"ID":"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d","Type":"ContainerDied","Data":"045840ae16d1320ba2581d951d411c009ee75e2a4c53cffd2438b4048e1addc2"}
Jul 18 17:23:00.839971 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.839840    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x" event={"ID":"f6ac8f08-9369-42a9-8f5d-838f0ba4d21d","Type":"ContainerDied","Data":"b71de1e87cd99214273984ed73c4dd7921977230431a3db9a0dc5e7e10a38db5"}
Jul 18 17:23:00.839971 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.839890    2554 scope.go:117] "RemoveContainer" containerID="045840ae16d1320ba2581d951d411c009ee75e2a4c53cffd2438b4048e1addc2"
Jul 18 17:23:00.855627 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.855605    2554 scope.go:117] "RemoveContainer" containerID="045840ae16d1320ba2581d951d411c009ee75e2a4c53cffd2438b4048e1addc2"
Jul 18 17:23:00.855902 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:23:00.855882    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045840ae16d1320ba2581d951d411c009ee75e2a4c53cffd2438b4048e1addc2\": container with ID starting with 045840ae16d1320ba2581d951d411c009ee75e2a4c53cffd2438b4048e1addc2 not found: ID does not exist" containerID="045840ae16d1320ba2581d951d411c009ee75e2a4c53cffd2438b4048e1addc2"
Jul 18 17:23:00.855983 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.855913    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045840ae16d1320ba2581d951d411c009ee75e2a4c53cffd2438b4048e1addc2"} err="failed to get container status \"045840ae16d1320ba2581d951d411c009ee75e2a4c53cffd2438b4048e1addc2\": rpc error: code = NotFound desc = could not find container \"045840ae16d1320ba2581d951d411c009ee75e2a4c53cffd2438b4048e1addc2\": container with ID starting with 045840ae16d1320ba2581d951d411c009ee75e2a4c53cffd2438b4048e1addc2 not found: ID does not exist"
Jul 18 17:23:00.865582 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.865557    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"]
Jul 18 17:23:00.868936 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:00.868914    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-497f21/auth-test-model-497f21-kserve-68d55457bc-67r4x"]
Jul 18 17:23:01.765662 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:01.765629    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9de71573-87fe-46c1-8cd2-53c32fd2e860" path="/var/lib/kubelet/pods/9de71573-87fe-46c1-8cd2-53c32fd2e860/volumes"
Jul 18 17:23:01.766066 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:01.766032    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6ac8f08-9369-42a9-8f5d-838f0ba4d21d" path="/var/lib/kubelet/pods/f6ac8f08-9369-42a9-8f5d-838f0ba4d21d/volumes"
Jul 18 17:23:03.772367 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.772330    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"]
Jul 18 17:23:03.772823 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.772650    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ac8f08-9369-42a9-8f5d-838f0ba4d21d" containerName="main"
Jul 18 17:23:03.772823 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.772660    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ac8f08-9369-42a9-8f5d-838f0ba4d21d" containerName="main"
Jul 18 17:23:03.772823 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.772676    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9de71573-87fe-46c1-8cd2-53c32fd2e860" containerName="main"
Jul 18 17:23:03.772823 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.772682    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9de71573-87fe-46c1-8cd2-53c32fd2e860" containerName="main"
Jul 18 17:23:03.772823 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.772741    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9de71573-87fe-46c1-8cd2-53c32fd2e860" containerName="main"
Jul 18 17:23:03.772823 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.772752    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6ac8f08-9369-42a9-8f5d-838f0ba4d21d" containerName="main"
Jul 18 17:23:03.777093 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.777076    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.780895 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.780872    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-78538d\"/\"default-dockercfg-r27rs\""
Jul 18 17:23:03.780895 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.780891    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-78538d\"/\"kube-root-ca.crt\""
Jul 18 17:23:03.781043 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.780929    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-78538d\"/\"openshift-service-ca.crt\""
Jul 18 17:23:03.781166 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.781150    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-78538d\"/\"sub-test-model-78538d-kserve-self-signed-certs\""
Jul 18 17:23:03.786420 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.786396    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"]
Jul 18 17:23:03.877380 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.877337    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-model-cache\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.877380 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.877381    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-tmp-dir\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.877629 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.877456    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-home\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.877629 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.877481    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-tls-certs\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.877629 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.877505    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smn8b\" (UniqueName: \"kubernetes.io/projected/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-kube-api-access-smn8b\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.877629 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.877608    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-dshm\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.922307 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.922278    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:23:03.978437 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.978408    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-model-cache\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.978624 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.978449    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-tmp-dir\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.978624 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.978495    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-home\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.978624 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.978515    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-tls-certs\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.978624 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.978532    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smn8b\" (UniqueName: \"kubernetes.io/projected/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-kube-api-access-smn8b\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.978624 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.978562    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-dshm\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.978934 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.978823    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-model-cache\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.978934 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.978907    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-home\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.979018 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.978973    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-tmp-dir\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.980915 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.980900    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-dshm\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.981235 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.981220    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-tls-certs\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:03.987608 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:03.987589    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smn8b\" (UniqueName: \"kubernetes.io/projected/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-kube-api-access-smn8b\") pod \"sub-test-model-78538d-kserve-748574fc9c-8zm2p\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") " pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:04.087196 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:04.087163    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:04.222952 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:04.222928    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"]
Jul 18 17:23:04.225120 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:23:04.225077    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fcd530e_75cf_47c7_8d39_4ad71f6b489e.slice/crio-f9d19ecb48749513a9191c08e4fda31b9e317501083e4390678ffb3a975c1b84 WatchSource:0}: Error finding container f9d19ecb48749513a9191c08e4fda31b9e317501083e4390678ffb3a975c1b84: Status 404 returned error can't find the container with id f9d19ecb48749513a9191c08e4fda31b9e317501083e4390678ffb3a975c1b84
Jul 18 17:23:04.861778 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:04.861741    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p" event={"ID":"4fcd530e-75cf-47c7-8d39-4ad71f6b489e","Type":"ContainerStarted","Data":"27a2dc3c277411c2b982d67349bdf4516f8536bb5fd0019ba459b744e62cad3b"}
Jul 18 17:23:04.862200 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:04.861787    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p" event={"ID":"4fcd530e-75cf-47c7-8d39-4ad71f6b489e","Type":"ContainerStarted","Data":"f9d19ecb48749513a9191c08e4fda31b9e317501083e4390678ffb3a975c1b84"}
Jul 18 17:23:04.880083 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:04.880034    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p" podStartSLOduration=1.880018093 podStartE2EDuration="1.880018093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:23:04.879203492 +0000 UTC m=+2803.577034176" watchObservedRunningTime="2026-07-18 17:23:04.880018093 +0000 UTC m=+2803.577848776"
Jul 18 17:23:14.087389 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:14.087346    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:14.087795 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:14.087410    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:14.099785 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:14.099751    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:14.921087 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:14.921057    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:23:47.896189 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:47.896152    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"]
Jul 18 17:23:47.899605 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:47.899589    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:47.903771 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:47.903750    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-497f21\"/\"kube-root-ca.crt\""
Jul 18 17:23:47.903771 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:47.903764    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-497f21\"/\"sub-test-model-497f21-kserve-self-signed-certs\""
Jul 18 17:23:47.904264 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:47.904249    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-497f21\"/\"openshift-service-ca.crt\""
Jul 18 17:23:47.904392 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:47.904374    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-497f21\"/\"default-dockercfg-dvrp4\""
Jul 18 17:23:47.916714 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:47.916691    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"]
Jul 18 17:23:47.948886 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:47.948861    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:23:48.060891 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.060858    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-dshm\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.061070 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.060928    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-tmp-dir\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.061070 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.060956    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-tls-certs\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.061070 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.061008    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2xpk\" (UniqueName: \"kubernetes.io/projected/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-kube-api-access-x2xpk\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.061070 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.061028    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-model-cache\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.061230 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.061092    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-home\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.161897 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.161789    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-tls-certs\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.161897 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.161871    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2xpk\" (UniqueName: \"kubernetes.io/projected/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-kube-api-access-x2xpk\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.162130 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.161903    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-model-cache\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.162130 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.161944    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-home\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.162130 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.161968    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-dshm\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.162130 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.162017    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-tmp-dir\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.162370 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.162345    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-home\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.162434 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.162357    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-model-cache\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.162490 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.162436    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-tmp-dir\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.164410 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.164386    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-dshm\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.164474 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.164452    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-tls-certs\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.171315 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.171293    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2xpk\" (UniqueName: \"kubernetes.io/projected/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-kube-api-access-x2xpk\") pod \"sub-test-model-497f21-kserve-5645bbc4bf-k2q45\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") " pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.209242 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.209214    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:48.345524 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:48.345499    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"]
Jul 18 17:23:48.347602 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:23:48.347576    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3ef8a47_fe45_4756_9abb_a731e5cabb7f.slice/crio-8a0690f5ca3cb2664a81b6f4b71a223ac8ab1653072729f1f6969ff41457071f WatchSource:0}: Error finding container 8a0690f5ca3cb2664a81b6f4b71a223ac8ab1653072729f1f6969ff41457071f: Status 404 returned error can't find the container with id 8a0690f5ca3cb2664a81b6f4b71a223ac8ab1653072729f1f6969ff41457071f
Jul 18 17:23:49.067060 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:49.067022    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45" event={"ID":"a3ef8a47-fe45-4756-9abb-a731e5cabb7f","Type":"ContainerStarted","Data":"1b3dbe4b40f7304162d7a88adf9ce09e90f7847d7585217a4fb2a55fe8cf3ba7"}
Jul 18 17:23:49.067478 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:49.067069    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45" event={"ID":"a3ef8a47-fe45-4756-9abb-a731e5cabb7f","Type":"ContainerStarted","Data":"8a0690f5ca3cb2664a81b6f4b71a223ac8ab1653072729f1f6969ff41457071f"}
Jul 18 17:23:49.092102 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:49.092057    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45" podStartSLOduration=2.0920386 podStartE2EDuration="2.0920386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:23:49.091879364 +0000 UTC m=+2847.789710046" watchObservedRunningTime="2026-07-18 17:23:49.0920386 +0000 UTC m=+2847.789869299"
Jul 18 17:23:51.511635 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:51.511554    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:23:57.918577 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:57.918539    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:23:58.019794 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:58.019748    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:23:58.210278 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:58.210192    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:58.210278 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:58.210243    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:58.222784 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:58.222756    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:23:59.124105 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:23:59.124080    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:24:22.129559 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:22.129473    2554 scope.go:117] "RemoveContainer" containerID="4384894b7abd9bb9f078db5220223ba9fc99aba2b92ab42dc264fe47774c935c"
Jul 18 17:24:31.318539 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:31.318508    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"]
Jul 18 17:24:31.348461 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:31.348431    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:24:32.320982 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:32.320949    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"]
Jul 18 17:24:32.340219 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:32.340179    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:24:39.099814 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:39.099768    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:24:39.202171 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:39.202140    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:24:46.338297 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.338255    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p" podUID="4fcd530e-75cf-47c7-8d39-4ad71f6b489e" containerName="main" containerID="cri-o://27a2dc3c277411c2b982d67349bdf4516f8536bb5fd0019ba459b744e62cad3b" gracePeriod=45
Jul 18 17:24:46.588972 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.588905    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:24:46.688414 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.688377    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-tmp-dir\") pod \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") "
Jul 18 17:24:46.688588 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.688420    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smn8b\" (UniqueName: \"kubernetes.io/projected/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-kube-api-access-smn8b\") pod \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") "
Jul 18 17:24:46.688588 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.688460    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-home\") pod \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") "
Jul 18 17:24:46.688588 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.688490    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-dshm\") pod \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") "
Jul 18 17:24:46.688588 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.688530    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-tls-certs\") pod \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") "
Jul 18 17:24:46.688588 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.688572    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-model-cache\") pod \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\" (UID: \"4fcd530e-75cf-47c7-8d39-4ad71f6b489e\") "
Jul 18 17:24:46.688887 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.688702    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fcd530e-75cf-47c7-8d39-4ad71f6b489e" (UID: "4fcd530e-75cf-47c7-8d39-4ad71f6b489e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:24:46.688887 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.688772    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-home" (OuterVolumeSpecName: "home") pod "4fcd530e-75cf-47c7-8d39-4ad71f6b489e" (UID: "4fcd530e-75cf-47c7-8d39-4ad71f6b489e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:24:46.689012 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.688951    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fcd530e-75cf-47c7-8d39-4ad71f6b489e" (UID: "4fcd530e-75cf-47c7-8d39-4ad71f6b489e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:24:46.689012 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.688962    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-tmp-dir\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:24:46.689012 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.688993    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-home\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:24:46.690808 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.690785    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-dshm" (OuterVolumeSpecName: "dshm") pod "4fcd530e-75cf-47c7-8d39-4ad71f6b489e" (UID: "4fcd530e-75cf-47c7-8d39-4ad71f6b489e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:24:46.690916 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.690877    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-kube-api-access-smn8b" (OuterVolumeSpecName: "kube-api-access-smn8b") pod "4fcd530e-75cf-47c7-8d39-4ad71f6b489e" (UID: "4fcd530e-75cf-47c7-8d39-4ad71f6b489e"). InnerVolumeSpecName "kube-api-access-smn8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:24:46.690975 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.690936    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fcd530e-75cf-47c7-8d39-4ad71f6b489e" (UID: "4fcd530e-75cf-47c7-8d39-4ad71f6b489e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:24:46.790313 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.790279    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-tls-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:24:46.790313 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.790307    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-model-cache\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:24:46.790313 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.790316    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smn8b\" (UniqueName: \"kubernetes.io/projected/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-kube-api-access-smn8b\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:24:46.790313 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:46.790324    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcd530e-75cf-47c7-8d39-4ad71f6b489e-dshm\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:24:47.341114 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.341080    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45" podUID="a3ef8a47-fe45-4756-9abb-a731e5cabb7f" containerName="main" containerID="cri-o://1b3dbe4b40f7304162d7a88adf9ce09e90f7847d7585217a4fb2a55fe8cf3ba7" gracePeriod=45
Jul 18 17:24:47.342689 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.342663    2554 generic.go:356] "Generic (PLEG): container finished" podID="4fcd530e-75cf-47c7-8d39-4ad71f6b489e" containerID="27a2dc3c277411c2b982d67349bdf4516f8536bb5fd0019ba459b744e62cad3b" exitCode=0
Jul 18 17:24:47.342800 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.342701    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p" event={"ID":"4fcd530e-75cf-47c7-8d39-4ad71f6b489e","Type":"ContainerDied","Data":"27a2dc3c277411c2b982d67349bdf4516f8536bb5fd0019ba459b744e62cad3b"}
Jul 18 17:24:47.342800 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.342719    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"
Jul 18 17:24:47.342800 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.342735    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p" event={"ID":"4fcd530e-75cf-47c7-8d39-4ad71f6b489e","Type":"ContainerDied","Data":"f9d19ecb48749513a9191c08e4fda31b9e317501083e4390678ffb3a975c1b84"}
Jul 18 17:24:47.342800 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.342752    2554 scope.go:117] "RemoveContainer" containerID="27a2dc3c277411c2b982d67349bdf4516f8536bb5fd0019ba459b744e62cad3b"
Jul 18 17:24:47.353980 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.353963    2554 scope.go:117] "RemoveContainer" containerID="27a2dc3c277411c2b982d67349bdf4516f8536bb5fd0019ba459b744e62cad3b"
Jul 18 17:24:47.354238 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:24:47.354218    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a2dc3c277411c2b982d67349bdf4516f8536bb5fd0019ba459b744e62cad3b\": container with ID starting with 27a2dc3c277411c2b982d67349bdf4516f8536bb5fd0019ba459b744e62cad3b not found: ID does not exist" containerID="27a2dc3c277411c2b982d67349bdf4516f8536bb5fd0019ba459b744e62cad3b"
Jul 18 17:24:47.354302 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.354246    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a2dc3c277411c2b982d67349bdf4516f8536bb5fd0019ba459b744e62cad3b"} err="failed to get container status \"27a2dc3c277411c2b982d67349bdf4516f8536bb5fd0019ba459b744e62cad3b\": rpc error: code = NotFound desc = could not find container \"27a2dc3c277411c2b982d67349bdf4516f8536bb5fd0019ba459b744e62cad3b\": container with ID starting with 27a2dc3c277411c2b982d67349bdf4516f8536bb5fd0019ba459b744e62cad3b not found: ID does not exist"
Jul 18 17:24:47.389582 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.389550    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:24:47.419759 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.419737    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"]
Jul 18 17:24:47.426985 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.426962    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-78538d/sub-test-model-78538d-kserve-748574fc9c-8zm2p"]
Jul 18 17:24:47.490060 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.490031    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:24:47.595383 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.595320    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:24:47.698471 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.698444    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-home\") pod \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") "
Jul 18 17:24:47.698662 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.698484    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2xpk\" (UniqueName: \"kubernetes.io/projected/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-kube-api-access-x2xpk\") pod \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") "
Jul 18 17:24:47.698662 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.698551    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-dshm\") pod \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") "
Jul 18 17:24:47.698662 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.698570    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-tmp-dir\") pod \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") "
Jul 18 17:24:47.698662 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.698588    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-model-cache\") pod \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") "
Jul 18 17:24:47.698875 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.698678    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-tls-certs\") pod \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\" (UID: \"a3ef8a47-fe45-4756-9abb-a731e5cabb7f\") "
Jul 18 17:24:47.698978 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.698684    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-home" (OuterVolumeSpecName: "home") pod "a3ef8a47-fe45-4756-9abb-a731e5cabb7f" (UID: "a3ef8a47-fe45-4756-9abb-a731e5cabb7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:24:47.699161 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.698831    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3ef8a47-fe45-4756-9abb-a731e5cabb7f" (UID: "a3ef8a47-fe45-4756-9abb-a731e5cabb7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:24:47.699161 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.698869    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3ef8a47-fe45-4756-9abb-a731e5cabb7f" (UID: "a3ef8a47-fe45-4756-9abb-a731e5cabb7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:24:47.700862 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.700826    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-kube-api-access-x2xpk" (OuterVolumeSpecName: "kube-api-access-x2xpk") pod "a3ef8a47-fe45-4756-9abb-a731e5cabb7f" (UID: "a3ef8a47-fe45-4756-9abb-a731e5cabb7f"). InnerVolumeSpecName "kube-api-access-x2xpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:24:47.700960 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.700828    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-dshm" (OuterVolumeSpecName: "dshm") pod "a3ef8a47-fe45-4756-9abb-a731e5cabb7f" (UID: "a3ef8a47-fe45-4756-9abb-a731e5cabb7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:24:47.701035 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.701012    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3ef8a47-fe45-4756-9abb-a731e5cabb7f" (UID: "a3ef8a47-fe45-4756-9abb-a731e5cabb7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:24:47.766691 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.766660    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fcd530e-75cf-47c7-8d39-4ad71f6b489e" path="/var/lib/kubelet/pods/4fcd530e-75cf-47c7-8d39-4ad71f6b489e/volumes"
Jul 18 17:24:47.799614 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.799582    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-dshm\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:24:47.799614 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.799604    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-tmp-dir\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:24:47.799614 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.799613    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-model-cache\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:24:47.799862 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.799623    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-tls-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:24:47.799862 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.799634    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-home\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:24:47.799862 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:47.799647    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2xpk\" (UniqueName: \"kubernetes.io/projected/a3ef8a47-fe45-4756-9abb-a731e5cabb7f-kube-api-access-x2xpk\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:24:48.348928 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:48.348892    2554 generic.go:356] "Generic (PLEG): container finished" podID="a3ef8a47-fe45-4756-9abb-a731e5cabb7f" containerID="1b3dbe4b40f7304162d7a88adf9ce09e90f7847d7585217a4fb2a55fe8cf3ba7" exitCode=0
Jul 18 17:24:48.349386 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:48.348952    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"
Jul 18 17:24:48.349386 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:48.348980    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45" event={"ID":"a3ef8a47-fe45-4756-9abb-a731e5cabb7f","Type":"ContainerDied","Data":"1b3dbe4b40f7304162d7a88adf9ce09e90f7847d7585217a4fb2a55fe8cf3ba7"}
Jul 18 17:24:48.349386 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:48.349026    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45" event={"ID":"a3ef8a47-fe45-4756-9abb-a731e5cabb7f","Type":"ContainerDied","Data":"8a0690f5ca3cb2664a81b6f4b71a223ac8ab1653072729f1f6969ff41457071f"}
Jul 18 17:24:48.349386 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:48.349049    2554 scope.go:117] "RemoveContainer" containerID="1b3dbe4b40f7304162d7a88adf9ce09e90f7847d7585217a4fb2a55fe8cf3ba7"
Jul 18 17:24:48.357606 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:48.357590    2554 scope.go:117] "RemoveContainer" containerID="1b3dbe4b40f7304162d7a88adf9ce09e90f7847d7585217a4fb2a55fe8cf3ba7"
Jul 18 17:24:48.357840 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:24:48.357823    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3dbe4b40f7304162d7a88adf9ce09e90f7847d7585217a4fb2a55fe8cf3ba7\": container with ID starting with 1b3dbe4b40f7304162d7a88adf9ce09e90f7847d7585217a4fb2a55fe8cf3ba7 not found: ID does not exist" containerID="1b3dbe4b40f7304162d7a88adf9ce09e90f7847d7585217a4fb2a55fe8cf3ba7"
Jul 18 17:24:48.357933 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:48.357864    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3dbe4b40f7304162d7a88adf9ce09e90f7847d7585217a4fb2a55fe8cf3ba7"} err="failed to get container status \"1b3dbe4b40f7304162d7a88adf9ce09e90f7847d7585217a4fb2a55fe8cf3ba7\": rpc error: code = NotFound desc = could not find container \"1b3dbe4b40f7304162d7a88adf9ce09e90f7847d7585217a4fb2a55fe8cf3ba7\": container with ID starting with 1b3dbe4b40f7304162d7a88adf9ce09e90f7847d7585217a4fb2a55fe8cf3ba7 not found: ID does not exist"
Jul 18 17:24:48.365425 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:48.365405    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"]
Jul 18 17:24:48.370918 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:48.370898    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-497f21/sub-test-model-497f21-kserve-5645bbc4bf-k2q45"]
Jul 18 17:24:49.765838 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:49.765804    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3ef8a47-fe45-4756-9abb-a731e5cabb7f" path="/var/lib/kubelet/pods/a3ef8a47-fe45-4756-9abb-a731e5cabb7f/volumes"
Jul 18 17:24:50.172914 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.172833    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"]
Jul 18 17:24:50.173290 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.173271    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3ef8a47-fe45-4756-9abb-a731e5cabb7f" containerName="main"
Jul 18 17:24:50.173381 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.173293    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3ef8a47-fe45-4756-9abb-a731e5cabb7f" containerName="main"
Jul 18 17:24:50.173381 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.173310    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fcd530e-75cf-47c7-8d39-4ad71f6b489e" containerName="main"
Jul 18 17:24:50.173381 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.173319    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fcd530e-75cf-47c7-8d39-4ad71f6b489e" containerName="main"
Jul 18 17:24:50.173533 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.173424    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fcd530e-75cf-47c7-8d39-4ad71f6b489e" containerName="main"
Jul 18 17:24:50.173533 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.173437    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3ef8a47-fe45-4756-9abb-a731e5cabb7f" containerName="main"
Jul 18 17:24:50.178024 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.178004    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.180889 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.180841    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-78538d\"/\"kube-root-ca.crt\""
Jul 18 17:24:50.181830 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.181811    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-78538d\"/\"openshift-service-ca.crt\""
Jul 18 17:24:50.181960 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.181902    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-78538d\"/\"default-dockercfg-r27rs\""
Jul 18 17:24:50.181960 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.181909    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-78538d\"/\"rate-test-model-78538d-kserve-self-signed-certs\""
Jul 18 17:24:50.187983 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.187962    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"]
Jul 18 17:24:50.205484 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.205453    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:24:50.220250 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.220226    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-dshm\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.220349 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.220278    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rdmn\" (UniqueName: \"kubernetes.io/projected/c7d28409-9804-4e29-93fc-e425a6bd8b0d-kube-api-access-7rdmn\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.220349 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.220337    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-model-cache\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.220453 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.220380    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d28409-9804-4e29-93fc-e425a6bd8b0d-tls-certs\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.220453 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.220410    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-home\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.220453 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.220442    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-tmp-dir\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.321016 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.320978    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rdmn\" (UniqueName: \"kubernetes.io/projected/c7d28409-9804-4e29-93fc-e425a6bd8b0d-kube-api-access-7rdmn\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.321016 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.321018    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-model-cache\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.321251 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.321045    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d28409-9804-4e29-93fc-e425a6bd8b0d-tls-certs\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.321251 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.321074    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-home\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.321251 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.321114    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-tmp-dir\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.321251 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.321166    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-dshm\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.321472 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.321443    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-model-cache\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.321536 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.321458    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-home\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.321536 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.321527    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-tmp-dir\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.323639 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.323618    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-dshm\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.323746 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.323617    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d28409-9804-4e29-93fc-e425a6bd8b0d-tls-certs\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.329460 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.329440    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rdmn\" (UniqueName: \"kubernetes.io/projected/c7d28409-9804-4e29-93fc-e425a6bd8b0d-kube-api-access-7rdmn\") pod \"rate-test-model-78538d-kserve-5cc598f8dc-gj24k\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") " pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.489161 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.489076    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:24:50.621990 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:50.621965    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"]
Jul 18 17:24:50.624004 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:24:50.623975    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7d28409_9804_4e29_93fc_e425a6bd8b0d.slice/crio-ae80d59ed32679cedd86a49f681b0a936af48871c1f0ddcad352073d54186b0a WatchSource:0}: Error finding container ae80d59ed32679cedd86a49f681b0a936af48871c1f0ddcad352073d54186b0a: Status 404 returned error can't find the container with id ae80d59ed32679cedd86a49f681b0a936af48871c1f0ddcad352073d54186b0a
Jul 18 17:24:51.366691 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:51.366660    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k" event={"ID":"c7d28409-9804-4e29-93fc-e425a6bd8b0d","Type":"ContainerStarted","Data":"f09c9b4c8d4262cb28c2ab54a5e40874588e17cd4836739131a08cb8689f31c3"}
Jul 18 17:24:51.366691 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:51.366698    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k" event={"ID":"c7d28409-9804-4e29-93fc-e425a6bd8b0d","Type":"ContainerStarted","Data":"ae80d59ed32679cedd86a49f681b0a936af48871c1f0ddcad352073d54186b0a"}
Jul 18 17:24:51.389080 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:24:51.389030    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k" podStartSLOduration=1.389015077 podStartE2EDuration="1.389015077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:24:51.387759322 +0000 UTC m=+2910.085590008" watchObservedRunningTime="2026-07-18 17:24:51.389015077 +0000 UTC m=+2910.086845760"
Jul 18 17:25:00.489866 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:00.489821    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:25:00.490349 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:00.489892    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:25:00.502181 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:00.502158    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:25:01.423978 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:01.423948    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:25:10.382269 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.382234    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"]
Jul 18 17:25:10.385553 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.385536    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.394469 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.394453    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-497f21\"/\"kube-root-ca.crt\""
Jul 18 17:25:10.395566 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.395545    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-497f21\"/\"rate-test-model-497f21-kserve-self-signed-certs\""
Jul 18 17:25:10.395674 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.395612    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-497f21\"/\"default-dockercfg-dvrp4\""
Jul 18 17:25:10.395674 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.395632    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-497f21\"/\"openshift-service-ca.crt\""
Jul 18 17:25:10.402239 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.402220    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"]
Jul 18 17:25:10.477354 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.477294    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:25:10.509642 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.509615    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-tls-certs\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.509802 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.509648    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5st8p\" (UniqueName: \"kubernetes.io/projected/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-kube-api-access-5st8p\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.509802 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.509729    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-model-cache\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.509802 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.509766    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-tmp-dir\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.509802 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.509796    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-home\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.509982 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.509824    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-dshm\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.610857 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.610822    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-model-cache\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.611051 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.610894    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-tmp-dir\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.611051 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.610931    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-home\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.611051 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.610960    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-dshm\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.611051 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.611024    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-tls-certs\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.611275 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.611053    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st8p\" (UniqueName: \"kubernetes.io/projected/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-kube-api-access-5st8p\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.611332 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.611280    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-model-cache\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.611393 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.611344    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-home\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.611393 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.611372    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-tmp-dir\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.613496 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.613461    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-dshm\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.613745 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.613719    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-tls-certs\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.619145 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.619124    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5st8p\" (UniqueName: \"kubernetes.io/projected/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-kube-api-access-5st8p\") pod \"rate-test-model-497f21-kserve-79b9c99dbc-vnspx\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") " pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.695265 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.695179    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:10.832389 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:10.832364    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"]
Jul 18 17:25:10.834606 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:25:10.834577    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cc7ac0f_91f4_443c_a0f5_727c68ac6a5b.slice/crio-abbace3b13dda4f55a82d8537ac9021c4adc75098046353a7f490a346fe85b86 WatchSource:0}: Error finding container abbace3b13dda4f55a82d8537ac9021c4adc75098046353a7f490a346fe85b86: Status 404 returned error can't find the container with id abbace3b13dda4f55a82d8537ac9021c4adc75098046353a7f490a346fe85b86
Jul 18 17:25:11.422802 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:11.422773    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:25:11.459019 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:11.458979    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx" event={"ID":"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b","Type":"ContainerStarted","Data":"85b48863c3717721091f08f09692e4748774c6b4a5fbea9da57039b6ae6771eb"}
Jul 18 17:25:11.459019 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:11.459020    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx" event={"ID":"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b","Type":"ContainerStarted","Data":"abbace3b13dda4f55a82d8537ac9021c4adc75098046353a7f490a346fe85b86"}
Jul 18 17:25:11.477222 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:11.477173    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx" podStartSLOduration=1.477158012 podStartE2EDuration="1.477158012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:25:11.47592707 +0000 UTC m=+2930.173757755" watchObservedRunningTime="2026-07-18 17:25:11.477158012 +0000 UTC m=+2930.174988724"
Jul 18 17:25:11.480964 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:11.480920    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:25:20.696160 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:20.696127    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:20.696160 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:20.696167    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:20.708798 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:20.708774    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:21.516317 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:21.516284    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:25:22.149019 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:22.148987    2554 scope.go:117] "RemoveContainer" containerID="199cfa1588ffa9000fcf9d001bdd173241c3a429c6d65653db141b9ba7241458"
Jul 18 17:25:51.094700 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:51.094625    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"]
Jul 18 17:25:51.117834 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:51.117803    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:25:52.042435 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:52.042401    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"]
Jul 18 17:25:52.066008 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:25:52.065966    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:06.114667 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.114601    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k" podUID="c7d28409-9804-4e29-93fc-e425a6bd8b0d" containerName="main" containerID="cri-o://f09c9b4c8d4262cb28c2ab54a5e40874588e17cd4836739131a08cb8689f31c3" gracePeriod=45
Jul 18 17:26:06.366167 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.366108    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:26:06.506766 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.506734    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rdmn\" (UniqueName: \"kubernetes.io/projected/c7d28409-9804-4e29-93fc-e425a6bd8b0d-kube-api-access-7rdmn\") pod \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") "
Jul 18 17:26:06.506766 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.506768    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-tmp-dir\") pod \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") "
Jul 18 17:26:06.507043 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.506795    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d28409-9804-4e29-93fc-e425a6bd8b0d-tls-certs\") pod \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") "
Jul 18 17:26:06.507043 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.506931    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-model-cache\") pod \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") "
Jul 18 17:26:06.507043 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.506968    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-home\") pod \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") "
Jul 18 17:26:06.507043 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.507012    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-dshm\") pod \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\" (UID: \"c7d28409-9804-4e29-93fc-e425a6bd8b0d\") "
Jul 18 17:26:06.507244 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.507074    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7d28409-9804-4e29-93fc-e425a6bd8b0d" (UID: "c7d28409-9804-4e29-93fc-e425a6bd8b0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:26:06.507340 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.507284    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-home" (OuterVolumeSpecName: "home") pod "c7d28409-9804-4e29-93fc-e425a6bd8b0d" (UID: "c7d28409-9804-4e29-93fc-e425a6bd8b0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:26:06.507340 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.507333    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-tmp-dir\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:06.507702 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.507355    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7d28409-9804-4e29-93fc-e425a6bd8b0d" (UID: "c7d28409-9804-4e29-93fc-e425a6bd8b0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:26:06.509153 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.509122    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-dshm" (OuterVolumeSpecName: "dshm") pod "c7d28409-9804-4e29-93fc-e425a6bd8b0d" (UID: "c7d28409-9804-4e29-93fc-e425a6bd8b0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:26:06.509244 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.509170    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d28409-9804-4e29-93fc-e425a6bd8b0d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7d28409-9804-4e29-93fc-e425a6bd8b0d" (UID: "c7d28409-9804-4e29-93fc-e425a6bd8b0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:26:06.509244 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.509193    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d28409-9804-4e29-93fc-e425a6bd8b0d-kube-api-access-7rdmn" (OuterVolumeSpecName: "kube-api-access-7rdmn") pod "c7d28409-9804-4e29-93fc-e425a6bd8b0d" (UID: "c7d28409-9804-4e29-93fc-e425a6bd8b0d"). InnerVolumeSpecName "kube-api-access-7rdmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:26:06.607991 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.607956    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-dshm\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:06.607991 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.607984    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rdmn\" (UniqueName: \"kubernetes.io/projected/c7d28409-9804-4e29-93fc-e425a6bd8b0d-kube-api-access-7rdmn\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:06.607991 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.607996    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d28409-9804-4e29-93fc-e425a6bd8b0d-tls-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:06.608229 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.608006    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-model-cache\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:06.608229 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.608014    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d28409-9804-4e29-93fc-e425a6bd8b0d-home\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:06.707620 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.707534    2554 generic.go:356] "Generic (PLEG): container finished" podID="c7d28409-9804-4e29-93fc-e425a6bd8b0d" containerID="f09c9b4c8d4262cb28c2ab54a5e40874588e17cd4836739131a08cb8689f31c3" exitCode=0
Jul 18 17:26:06.707620 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.707578    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"
Jul 18 17:26:06.707886 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.707628    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k" event={"ID":"c7d28409-9804-4e29-93fc-e425a6bd8b0d","Type":"ContainerDied","Data":"f09c9b4c8d4262cb28c2ab54a5e40874588e17cd4836739131a08cb8689f31c3"}
Jul 18 17:26:06.707886 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.707672    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k" event={"ID":"c7d28409-9804-4e29-93fc-e425a6bd8b0d","Type":"ContainerDied","Data":"ae80d59ed32679cedd86a49f681b0a936af48871c1f0ddcad352073d54186b0a"}
Jul 18 17:26:06.707886 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.707701    2554 scope.go:117] "RemoveContainer" containerID="f09c9b4c8d4262cb28c2ab54a5e40874588e17cd4836739131a08cb8689f31c3"
Jul 18 17:26:06.716885 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.716864    2554 scope.go:117] "RemoveContainer" containerID="f09c9b4c8d4262cb28c2ab54a5e40874588e17cd4836739131a08cb8689f31c3"
Jul 18 17:26:06.717139 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:26:06.717122    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f09c9b4c8d4262cb28c2ab54a5e40874588e17cd4836739131a08cb8689f31c3\": container with ID starting with f09c9b4c8d4262cb28c2ab54a5e40874588e17cd4836739131a08cb8689f31c3 not found: ID does not exist" containerID="f09c9b4c8d4262cb28c2ab54a5e40874588e17cd4836739131a08cb8689f31c3"
Jul 18 17:26:06.717190 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.717146    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09c9b4c8d4262cb28c2ab54a5e40874588e17cd4836739131a08cb8689f31c3"} err="failed to get container status \"f09c9b4c8d4262cb28c2ab54a5e40874588e17cd4836739131a08cb8689f31c3\": rpc error: code = NotFound desc = could not find container \"f09c9b4c8d4262cb28c2ab54a5e40874588e17cd4836739131a08cb8689f31c3\": container with ID starting with f09c9b4c8d4262cb28c2ab54a5e40874588e17cd4836739131a08cb8689f31c3 not found: ID does not exist"
Jul 18 17:26:06.728515 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.728485    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"]
Jul 18 17:26:06.732552 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:06.732532    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-78538d/rate-test-model-78538d-kserve-5cc598f8dc-gj24k"]
Jul 18 17:26:07.062757 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.062672    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx" podUID="0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b" containerName="main" containerID="cri-o://85b48863c3717721091f08f09692e4748774c6b4a5fbea9da57039b6ae6771eb" gracePeriod=45
Jul 18 17:26:07.310312 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.310292    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:26:07.415982 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.415947    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-tmp-dir\") pod \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") "
Jul 18 17:26:07.415982 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.415981    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-tls-certs\") pod \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") "
Jul 18 17:26:07.416231 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.416021    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5st8p\" (UniqueName: \"kubernetes.io/projected/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-kube-api-access-5st8p\") pod \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") "
Jul 18 17:26:07.416231 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.416044    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-home\") pod \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") "
Jul 18 17:26:07.416231 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.416071    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-model-cache\") pod \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") "
Jul 18 17:26:07.416231 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.416107    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-dshm\") pod \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\" (UID: \"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b\") "
Jul 18 17:26:07.416440 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.416296    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b" (UID: "0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:26:07.416440 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.416352    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-home" (OuterVolumeSpecName: "home") pod "0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b" (UID: "0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:26:07.416440 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.416365    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b" (UID: "0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:26:07.416565 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.416466    2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-tmp-dir\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:07.416565 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.416486    2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-home\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:07.418279 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.418254    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b" (UID: "0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:26:07.418376 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.418299    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-kube-api-access-5st8p" (OuterVolumeSpecName: "kube-api-access-5st8p") pod "0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b" (UID: "0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b"). InnerVolumeSpecName "kube-api-access-5st8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:26:07.418457 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.418442    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-dshm" (OuterVolumeSpecName: "dshm") pod "0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b" (UID: "0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:26:07.517207 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.517173    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-model-cache\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:07.517207 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.517201    2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-dshm\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:07.517382 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.517216    2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-tls-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:07.517382 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.517228    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5st8p\" (UniqueName: \"kubernetes.io/projected/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b-kube-api-access-5st8p\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:07.715252 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.715169    2554 generic.go:356] "Generic (PLEG): container finished" podID="0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b" containerID="85b48863c3717721091f08f09692e4748774c6b4a5fbea9da57039b6ae6771eb" exitCode=0
Jul 18 17:26:07.715252 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.715231    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"
Jul 18 17:26:07.715252 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.715239    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx" event={"ID":"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b","Type":"ContainerDied","Data":"85b48863c3717721091f08f09692e4748774c6b4a5fbea9da57039b6ae6771eb"}
Jul 18 17:26:07.715482 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.715276    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx" event={"ID":"0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b","Type":"ContainerDied","Data":"abbace3b13dda4f55a82d8537ac9021c4adc75098046353a7f490a346fe85b86"}
Jul 18 17:26:07.715482 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.715298    2554 scope.go:117] "RemoveContainer" containerID="85b48863c3717721091f08f09692e4748774c6b4a5fbea9da57039b6ae6771eb"
Jul 18 17:26:07.724368 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.724346    2554 scope.go:117] "RemoveContainer" containerID="85b48863c3717721091f08f09692e4748774c6b4a5fbea9da57039b6ae6771eb"
Jul 18 17:26:07.724626 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:26:07.724599    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b48863c3717721091f08f09692e4748774c6b4a5fbea9da57039b6ae6771eb\": container with ID starting with 85b48863c3717721091f08f09692e4748774c6b4a5fbea9da57039b6ae6771eb not found: ID does not exist" containerID="85b48863c3717721091f08f09692e4748774c6b4a5fbea9da57039b6ae6771eb"
Jul 18 17:26:07.724724 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.724634    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b48863c3717721091f08f09692e4748774c6b4a5fbea9da57039b6ae6771eb"} err="failed to get container status \"85b48863c3717721091f08f09692e4748774c6b4a5fbea9da57039b6ae6771eb\": rpc error: code = NotFound desc = could not find container \"85b48863c3717721091f08f09692e4748774c6b4a5fbea9da57039b6ae6771eb\": container with ID starting with 85b48863c3717721091f08f09692e4748774c6b4a5fbea9da57039b6ae6771eb not found: ID does not exist"
Jul 18 17:26:07.736267 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.736243    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"]
Jul 18 17:26:07.744610 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.744580    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-497f21/rate-test-model-497f21-kserve-79b9c99dbc-vnspx"]
Jul 18 17:26:07.767659 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.767627    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b" path="/var/lib/kubelet/pods/0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b/volumes"
Jul 18 17:26:07.768335 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:07.768318    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d28409-9804-4e29-93fc-e425a6bd8b0d" path="/var/lib/kubelet/pods/c7d28409-9804-4e29-93fc-e425a6bd8b0d/volumes"
Jul 18 17:26:11.516901 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:11.516861    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:11.620489 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:11.620453    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:21.337648 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.337603    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"]
Jul 18 17:26:21.338213 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.337876    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56" podUID="3fe8387d-8eed-47a7-a939-75ab5c6aecb6" containerName="maas-api" containerID="cri-o://34955f19c58939c5aa9c1ec75b133c8f538df9380a2359f1912d26cd65f64482" gracePeriod=30
Jul 18 17:26:21.369603 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.369578    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:21.584938 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.584913    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"
Jul 18 17:26:21.634422 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.634352    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-maas-api-tls\") pod \"3fe8387d-8eed-47a7-a939-75ab5c6aecb6\" (UID: \"3fe8387d-8eed-47a7-a939-75ab5c6aecb6\") "
Jul 18 17:26:21.634422 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.634387    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vkzj\" (UniqueName: \"kubernetes.io/projected/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-kube-api-access-4vkzj\") pod \"3fe8387d-8eed-47a7-a939-75ab5c6aecb6\" (UID: \"3fe8387d-8eed-47a7-a939-75ab5c6aecb6\") "
Jul 18 17:26:21.634592 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.634441    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-openshift-service-ca\") pod \"3fe8387d-8eed-47a7-a939-75ab5c6aecb6\" (UID: \"3fe8387d-8eed-47a7-a939-75ab5c6aecb6\") "
Jul 18 17:26:21.634949 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.634922    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3fe8387d-8eed-47a7-a939-75ab5c6aecb6" (UID: "3fe8387d-8eed-47a7-a939-75ab5c6aecb6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:26:21.636644 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.636613    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3fe8387d-8eed-47a7-a939-75ab5c6aecb6" (UID: "3fe8387d-8eed-47a7-a939-75ab5c6aecb6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:26:21.636742 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.636714    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-kube-api-access-4vkzj" (OuterVolumeSpecName: "kube-api-access-4vkzj") pod "3fe8387d-8eed-47a7-a939-75ab5c6aecb6" (UID: "3fe8387d-8eed-47a7-a939-75ab5c6aecb6"). InnerVolumeSpecName "kube-api-access-4vkzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:26:21.736078 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.736047    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-maas-api-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:21.736078 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.736075    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vkzj\" (UniqueName: \"kubernetes.io/projected/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-kube-api-access-4vkzj\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:21.736078 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.736087    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fe8387d-8eed-47a7-a939-75ab5c6aecb6-openshift-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:21.786879 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.786824    2554 generic.go:356] "Generic (PLEG): container finished" podID="3fe8387d-8eed-47a7-a939-75ab5c6aecb6" containerID="34955f19c58939c5aa9c1ec75b133c8f538df9380a2359f1912d26cd65f64482" exitCode=0
Jul 18 17:26:21.787047 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.786899    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"
Jul 18 17:26:21.787047 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.786910    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56" event={"ID":"3fe8387d-8eed-47a7-a939-75ab5c6aecb6","Type":"ContainerDied","Data":"34955f19c58939c5aa9c1ec75b133c8f538df9380a2359f1912d26cd65f64482"}
Jul 18 17:26:21.787047 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.786996    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56" event={"ID":"3fe8387d-8eed-47a7-a939-75ab5c6aecb6","Type":"ContainerDied","Data":"4451547b8840bd9d76238b7e46ee2b0fe5b35e145f1408273a945649748fefaf"}
Jul 18 17:26:21.787047 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.787024    2554 scope.go:117] "RemoveContainer" containerID="34955f19c58939c5aa9c1ec75b133c8f538df9380a2359f1912d26cd65f64482"
Jul 18 17:26:21.795611 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.795594    2554 scope.go:117] "RemoveContainer" containerID="34955f19c58939c5aa9c1ec75b133c8f538df9380a2359f1912d26cd65f64482"
Jul 18 17:26:21.795912 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:26:21.795893    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34955f19c58939c5aa9c1ec75b133c8f538df9380a2359f1912d26cd65f64482\": container with ID starting with 34955f19c58939c5aa9c1ec75b133c8f538df9380a2359f1912d26cd65f64482 not found: ID does not exist" containerID="34955f19c58939c5aa9c1ec75b133c8f538df9380a2359f1912d26cd65f64482"
Jul 18 17:26:21.795965 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.795920    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34955f19c58939c5aa9c1ec75b133c8f538df9380a2359f1912d26cd65f64482"} err="failed to get container status \"34955f19c58939c5aa9c1ec75b133c8f538df9380a2359f1912d26cd65f64482\": rpc error: code = NotFound desc = could not find container \"34955f19c58939c5aa9c1ec75b133c8f538df9380a2359f1912d26cd65f64482\": container with ID starting with 34955f19c58939c5aa9c1ec75b133c8f538df9380a2359f1912d26cd65f64482 not found: ID does not exist"
Jul 18 17:26:21.806744 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.806724    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"]
Jul 18 17:26:21.810572 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:21.810555    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-78538d-757dd4bd6d-z4g56"]
Jul 18 17:26:23.765976 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:23.765939    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fe8387d-8eed-47a7-a939-75ab5c6aecb6" path="/var/lib/kubelet/pods/3fe8387d-8eed-47a7-a939-75ab5c6aecb6/volumes"
Jul 18 17:26:25.978653 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:25.978616    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:26.354584 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:26.354552    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:32.434559 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:32.434521    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:32.542503 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:32.542461    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:32.670562 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:32.670523    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"]
Jul 18 17:26:32.670765 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:32.670744    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45" podUID="afec0482-b062-4758-b518-850636637cf1" containerName="istio-proxy" containerID="cri-o://33a0d083d3b503ee33ca5e52f1fc70485e10574eeac4833364ce7cacafd26219" gracePeriod=30
Jul 18 17:26:32.684534 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:32.684509    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:32.908934 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:32.908901    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:33.028972 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.028938    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx"]
Jul 18 17:26:33.029274 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.029262    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b" containerName="main"
Jul 18 17:26:33.029324 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.029275    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b" containerName="main"
Jul 18 17:26:33.029324 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.029291    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe8387d-8eed-47a7-a939-75ab5c6aecb6" containerName="maas-api"
Jul 18 17:26:33.029324 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.029296    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe8387d-8eed-47a7-a939-75ab5c6aecb6" containerName="maas-api"
Jul 18 17:26:33.029324 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.029314    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d28409-9804-4e29-93fc-e425a6bd8b0d" containerName="main"
Jul 18 17:26:33.029324 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.029319    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d28409-9804-4e29-93fc-e425a6bd8b0d" containerName="main"
Jul 18 17:26:33.029507 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.029367    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe8387d-8eed-47a7-a939-75ab5c6aecb6" containerName="maas-api"
Jul 18 17:26:33.029507 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.029376    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cc7ac0f-91f4-443c-a0f5-727c68ac6a5b" containerName="main"
Jul 18 17:26:33.029507 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.029386    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d28409-9804-4e29-93fc-e425a6bd8b0d" containerName="main"
Jul 18 17:26:33.032121 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.032103    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx"
Jul 18 17:26:33.034557 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.034535    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7rq8\""
Jul 18 17:26:33.041190 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.041156    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx"]
Jul 18 17:26:33.136388 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.136349    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx\" (UID: \"0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx"
Jul 18 17:26:33.237811 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.237724    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx\" (UID: \"0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx"
Jul 18 17:26:33.238322 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.238303    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx\" (UID: \"0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx"
Jul 18 17:26:33.343636 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.343598    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx"
Jul 18 17:26:33.474202 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.474181    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx"]
Jul 18 17:26:33.476353 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:26:33.476322    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6b64bc_f9a8_4f2b_bf6e_49b5008fa6f1.slice/crio-4d684e8fa8aa890e83818596a9a0b015f9fd582ea0fe108e7a9740113d319619 WatchSource:0}: Error finding container 4d684e8fa8aa890e83818596a9a0b015f9fd582ea0fe108e7a9740113d319619: Status 404 returned error can't find the container with id 4d684e8fa8aa890e83818596a9a0b015f9fd582ea0fe108e7a9740113d319619
Jul 18 17:26:33.537568 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.537535    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:33.639213 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.639179    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:33.839672 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.839635    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx" event={"ID":"0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1","Type":"ContainerStarted","Data":"f256b736d8a8ced00204c0f2aa4e9c1e32d02b9897cdf4ce3540480a7c821766"}
Jul 18 17:26:33.839867 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.839691    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx" event={"ID":"0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1","Type":"ContainerStarted","Data":"4d684e8fa8aa890e83818596a9a0b015f9fd582ea0fe108e7a9740113d319619"}
Jul 18 17:26:33.857019 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:33.856973    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx" podStartSLOduration=0.856957925 podStartE2EDuration="856.957925ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:26:33.855198314 +0000 UTC m=+3012.553028999" watchObservedRunningTime="2026-07-18 17:26:33.856957925 +0000 UTC m=+3012.554788608"
Jul 18 17:26:34.845350 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:34.845318    2554 generic.go:356] "Generic (PLEG): container finished" podID="0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1" containerID="f256b736d8a8ced00204c0f2aa4e9c1e32d02b9897cdf4ce3540480a7c821766" exitCode=0
Jul 18 17:26:34.845777 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:34.845404    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx" event={"ID":"0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1","Type":"ContainerDied","Data":"f256b736d8a8ced00204c0f2aa4e9c1e32d02b9897cdf4ce3540480a7c821766"}
Jul 18 17:26:35.975422 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:35.975396    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx"
Jul 18 17:26:36.064325 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:36.064290    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1-maas-api-service-ca\") pod \"0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1\" (UID: \"0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1\") "
Jul 18 17:26:36.064664 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:36.064643    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1" (UID: "0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:26:36.165486 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:36.165397    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1-maas-api-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:36.855766 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:36.855734    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx"
Jul 18 17:26:36.855958 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:36.855769    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx" event={"ID":"0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1","Type":"ContainerDied","Data":"4d684e8fa8aa890e83818596a9a0b015f9fd582ea0fe108e7a9740113d319619"}
Jul 18 17:26:36.855958 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:36.855813    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d684e8fa8aa890e83818596a9a0b015f9fd582ea0fe108e7a9740113d319619"
Jul 18 17:26:37.864123 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:37.864087    2554 generic.go:356] "Generic (PLEG): container finished" podID="afec0482-b062-4758-b518-850636637cf1" containerID="33a0d083d3b503ee33ca5e52f1fc70485e10574eeac4833364ce7cacafd26219" exitCode=0
Jul 18 17:26:37.864576 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:37.864138    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45" event={"ID":"afec0482-b062-4758-b518-850636637cf1","Type":"ContainerDied","Data":"33a0d083d3b503ee33ca5e52f1fc70485e10574eeac4833364ce7cacafd26219"}
Jul 18 17:26:37.864576 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:37.864198    2554 scope.go:117] "RemoveContainer" containerID="63125580a827dc2da7ba439b321cdc13bfb9104af2d3d9dce39b32926edf9e79"
Jul 18 17:26:37.936704 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:37.936633    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:26:38.082900 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.082869    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-istio-envoy\") pod \"afec0482-b062-4758-b518-850636637cf1\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") "
Jul 18 17:26:38.083089 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.082916    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-workload-certs\") pod \"afec0482-b062-4758-b518-850636637cf1\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") "
Jul 18 17:26:38.083089 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.082946    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afec0482-b062-4758-b518-850636637cf1-istio-token\") pod \"afec0482-b062-4758-b518-850636637cf1\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") "
Jul 18 17:26:38.083089 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.082970    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afec0482-b062-4758-b518-850636637cf1-istiod-ca-cert\") pod \"afec0482-b062-4758-b518-850636637cf1\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") "
Jul 18 17:26:38.083089 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.082989    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-istio-data\") pod \"afec0482-b062-4758-b518-850636637cf1\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") "
Jul 18 17:26:38.083089 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.083018    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-credential-socket\") pod \"afec0482-b062-4758-b518-850636637cf1\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") "
Jul 18 17:26:38.083359 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.083253    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afec0482-b062-4758-b518-850636637cf1-istio-podinfo\") pod \"afec0482-b062-4758-b518-850636637cf1\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") "
Jul 18 17:26:38.083359 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.083285    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "afec0482-b062-4758-b518-850636637cf1" (UID: "afec0482-b062-4758-b518-850636637cf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:26:38.083359 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.083295    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "afec0482-b062-4758-b518-850636637cf1" (UID: "afec0482-b062-4758-b518-850636637cf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:26:38.083359 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.083335    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4mvd\" (UniqueName: \"kubernetes.io/projected/afec0482-b062-4758-b518-850636637cf1-kube-api-access-h4mvd\") pod \"afec0482-b062-4758-b518-850636637cf1\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") "
Jul 18 17:26:38.083544 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.083390    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afec0482-b062-4758-b518-850636637cf1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "afec0482-b062-4758-b518-850636637cf1" (UID: "afec0482-b062-4758-b518-850636637cf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:26:38.083544 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.083404    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-workload-socket\") pod \"afec0482-b062-4758-b518-850636637cf1\" (UID: \"afec0482-b062-4758-b518-850636637cf1\") "
Jul 18 17:26:38.083678 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.083662    2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afec0482-b062-4758-b518-850636637cf1-istiod-ca-cert\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:38.083742 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.083683    2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-credential-socket\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:38.083742 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.083696    2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-workload-certs\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:38.083742 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.083720    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "afec0482-b062-4758-b518-850636637cf1" (UID: "afec0482-b062-4758-b518-850636637cf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:26:38.083964 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.083939    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-istio-data" (OuterVolumeSpecName: "istio-data") pod "afec0482-b062-4758-b518-850636637cf1" (UID: "afec0482-b062-4758-b518-850636637cf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:26:38.085276 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.085254    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "afec0482-b062-4758-b518-850636637cf1" (UID: "afec0482-b062-4758-b518-850636637cf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 18 17:26:38.085725 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.085706    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afec0482-b062-4758-b518-850636637cf1-kube-api-access-h4mvd" (OuterVolumeSpecName: "kube-api-access-h4mvd") pod "afec0482-b062-4758-b518-850636637cf1" (UID: "afec0482-b062-4758-b518-850636637cf1"). InnerVolumeSpecName "kube-api-access-h4mvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:26:38.085896 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.085877    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afec0482-b062-4758-b518-850636637cf1-istio-token" (OuterVolumeSpecName: "istio-token") pod "afec0482-b062-4758-b518-850636637cf1" (UID: "afec0482-b062-4758-b518-850636637cf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:26:38.085951 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.085877    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afec0482-b062-4758-b518-850636637cf1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "afec0482-b062-4758-b518-850636637cf1" (UID: "afec0482-b062-4758-b518-850636637cf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 18 17:26:38.184643 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.184610    2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-workload-socket\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:38.184643 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.184635    2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-istio-envoy\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:38.184643 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.184644    2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afec0482-b062-4758-b518-850636637cf1-istio-token\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:38.184643 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.184653    2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afec0482-b062-4758-b518-850636637cf1-istio-data\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:38.184927 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.184660    2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afec0482-b062-4758-b518-850636637cf1-istio-podinfo\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:38.184927 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.184668    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4mvd\" (UniqueName: \"kubernetes.io/projected/afec0482-b062-4758-b518-850636637cf1-kube-api-access-h4mvd\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:38.869939 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.869890    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45" event={"ID":"afec0482-b062-4758-b518-850636637cf1","Type":"ContainerDied","Data":"7ebc2b5c06f6fa14f82e15f81b5aba8707c7d4309360b8d68b108bdc9bfca749"}
Jul 18 17:26:38.870377 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.869963    2554 scope.go:117] "RemoveContainer" containerID="33a0d083d3b503ee33ca5e52f1fc70485e10574eeac4833364ce7cacafd26219"
Jul 18 17:26:38.870377 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.869910    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"
Jul 18 17:26:38.894420 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.894394    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"]
Jul 18 17:26:38.898089 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:38.898060    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-78538d-openshift-default-8c795f47-6jg45"]
Jul 18 17:26:39.765962 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:39.765934    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afec0482-b062-4758-b518-850636637cf1" path="/var/lib/kubelet/pods/afec0482-b062-4758-b518-850636637cf1/volumes"
Jul 18 17:26:43.425439 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.425407    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"]
Jul 18 17:26:43.425922 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.425666    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p" podUID="a38bdb90-a757-4169-8864-d08232ce0efe" containerName="maas-api" containerID="cri-o://f6b8d3800bae15bc21f87b68652c4d666bd1dd2499dda43110719b78818ebc06" gracePeriod=30
Jul 18 17:26:43.452440 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.452401    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:43.872181 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.872157    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"
Jul 18 17:26:43.893909 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.893867    2554 generic.go:356] "Generic (PLEG): container finished" podID="a38bdb90-a757-4169-8864-d08232ce0efe" containerID="f6b8d3800bae15bc21f87b68652c4d666bd1dd2499dda43110719b78818ebc06" exitCode=0
Jul 18 17:26:43.893909 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.893906    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"
Jul 18 17:26:43.894129 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.893940    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p" event={"ID":"a38bdb90-a757-4169-8864-d08232ce0efe","Type":"ContainerDied","Data":"f6b8d3800bae15bc21f87b68652c4d666bd1dd2499dda43110719b78818ebc06"}
Jul 18 17:26:43.894129 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.894063    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p" event={"ID":"a38bdb90-a757-4169-8864-d08232ce0efe","Type":"ContainerDied","Data":"9c763fbfdb127d848f147b6b84ba3a6dd3e3c51884d8ce6ad478c6d4cc71928c"}
Jul 18 17:26:43.894129 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.894096    2554 scope.go:117] "RemoveContainer" containerID="f6b8d3800bae15bc21f87b68652c4d666bd1dd2499dda43110719b78818ebc06"
Jul 18 17:26:43.902746 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.902726    2554 scope.go:117] "RemoveContainer" containerID="f6b8d3800bae15bc21f87b68652c4d666bd1dd2499dda43110719b78818ebc06"
Jul 18 17:26:43.903048 ip-10-0-142-77 kubenswrapper[2554]: E0718 17:26:43.903024    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b8d3800bae15bc21f87b68652c4d666bd1dd2499dda43110719b78818ebc06\": container with ID starting with f6b8d3800bae15bc21f87b68652c4d666bd1dd2499dda43110719b78818ebc06 not found: ID does not exist" containerID="f6b8d3800bae15bc21f87b68652c4d666bd1dd2499dda43110719b78818ebc06"
Jul 18 17:26:43.903143 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.903055    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b8d3800bae15bc21f87b68652c4d666bd1dd2499dda43110719b78818ebc06"} err="failed to get container status \"f6b8d3800bae15bc21f87b68652c4d666bd1dd2499dda43110719b78818ebc06\": rpc error: code = NotFound desc = could not find container \"f6b8d3800bae15bc21f87b68652c4d666bd1dd2499dda43110719b78818ebc06\": container with ID starting with f6b8d3800bae15bc21f87b68652c4d666bd1dd2499dda43110719b78818ebc06 not found: ID does not exist"
Jul 18 17:26:43.934282 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.934257    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn2dc\" (UniqueName: \"kubernetes.io/projected/a38bdb90-a757-4169-8864-d08232ce0efe-kube-api-access-tn2dc\") pod \"a38bdb90-a757-4169-8864-d08232ce0efe\" (UID: \"a38bdb90-a757-4169-8864-d08232ce0efe\") "
Jul 18 17:26:43.934379 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.934344    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a38bdb90-a757-4169-8864-d08232ce0efe-maas-api-tls\") pod \"a38bdb90-a757-4169-8864-d08232ce0efe\" (UID: \"a38bdb90-a757-4169-8864-d08232ce0efe\") "
Jul 18 17:26:43.934420 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.934389    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38bdb90-a757-4169-8864-d08232ce0efe-openshift-service-ca\") pod \"a38bdb90-a757-4169-8864-d08232ce0efe\" (UID: \"a38bdb90-a757-4169-8864-d08232ce0efe\") "
Jul 18 17:26:43.934763 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.934738    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38bdb90-a757-4169-8864-d08232ce0efe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a38bdb90-a757-4169-8864-d08232ce0efe" (UID: "a38bdb90-a757-4169-8864-d08232ce0efe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 18 17:26:43.936328 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.936301    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38bdb90-a757-4169-8864-d08232ce0efe-kube-api-access-tn2dc" (OuterVolumeSpecName: "kube-api-access-tn2dc") pod "a38bdb90-a757-4169-8864-d08232ce0efe" (UID: "a38bdb90-a757-4169-8864-d08232ce0efe"). InnerVolumeSpecName "kube-api-access-tn2dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 18 17:26:43.936415 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:43.936392    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a38bdb90-a757-4169-8864-d08232ce0efe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a38bdb90-a757-4169-8864-d08232ce0efe" (UID: "a38bdb90-a757-4169-8864-d08232ce0efe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 18 17:26:44.035585 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:44.035510    2554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38bdb90-a757-4169-8864-d08232ce0efe-openshift-service-ca\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:44.035585 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:44.035537    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn2dc\" (UniqueName: \"kubernetes.io/projected/a38bdb90-a757-4169-8864-d08232ce0efe-kube-api-access-tn2dc\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:44.035585 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:44.035547    2554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a38bdb90-a757-4169-8864-d08232ce0efe-maas-api-tls\") on node \"ip-10-0-142-77.ec2.internal\" DevicePath \"\""
Jul 18 17:26:44.236917 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:44.236887    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"]
Jul 18 17:26:44.240839 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:44.240814    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-497f21-54c54dbcbc-spv7p"]
Jul 18 17:26:45.767183 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:45.767152    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a38bdb90-a757-4169-8864-d08232ce0efe" path="/var/lib/kubelet/pods/a38bdb90-a757-4169-8864-d08232ce0efe/volumes"
Jul 18 17:26:48.166874 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:48.166828    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx"]
Jul 18 17:26:48.169687 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:48.169666    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-497f21-mkjrx"]
Jul 18 17:26:48.187760 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:48.187691    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:48.519318 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:48.519213    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:49.765652 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:49.765623    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1" path="/var/lib/kubelet/pods/0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1/volumes"
Jul 18 17:26:54.348247 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:54.348217    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:54.464429 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:54.464349    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:54.643219 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:54.643182    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:54.876197 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:54.876166    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:26:55.278192 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:26:55.278120    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d86c54866-njv76_92cf2d68-2200-4256-a36b-0465b913a01f/authorino/0.log"
Jul 18 17:27:01.299956 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:01.299921    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c768ddb5d-whpnh_7af4d913-b0bd-4af3-92e2-2ae1ad9debf4/copy-manifests/0.log"
Jul 18 17:27:01.312693 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:01.312662    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c768ddb5d-whpnh_7af4d913-b0bd-4af3-92e2-2ae1ad9debf4/manager/0.log"
Jul 18 17:27:02.042089 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:02.042061    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29739915-ffhzg_ce5a6238-5faa-4620-8182-98e628c6d47c/cleanup/0.log"
Jul 18 17:27:03.440016 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:03.437498    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d86c54866-njv76_92cf2d68-2200-4256-a36b-0465b913a01f/authorino/0.log"
Jul 18 17:27:03.551801 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:03.551740    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-9sqhw_53346a72-5b96-4904-8537-e16d68b3d7f6/manager/0.log"
Jul 18 17:27:04.941281 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:04.941223    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c475ff8c-hb9q5_40d62415-b2ff-4eb9-9b5b-ec90c705184b/kube-auth-proxy/0.log"
Jul 18 17:27:06.095279 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:06.095248    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f24hff_46f878d9-0b44-46e7-abea-3550442073ad/main/0.log"
Jul 18 17:27:06.316544 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:06.316511    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-z88hz_59d53068-56b7-4231-be50-ad6fa66a1f7d/main/0.log"
Jul 18 17:27:11.689291 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.689258    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fc66r/must-gather-hj8lj"]
Jul 18 17:27:11.689675 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.689598    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38bdb90-a757-4169-8864-d08232ce0efe" containerName="maas-api"
Jul 18 17:27:11.689675 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.689610    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38bdb90-a757-4169-8864-d08232ce0efe" containerName="maas-api"
Jul 18 17:27:11.689675 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.689618    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1" containerName="revoke-keys"
Jul 18 17:27:11.689675 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.689624    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1" containerName="revoke-keys"
Jul 18 17:27:11.689675 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.689634    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afec0482-b062-4758-b518-850636637cf1" containerName="istio-proxy"
Jul 18 17:27:11.689675 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.689639    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="afec0482-b062-4758-b518-850636637cf1" containerName="istio-proxy"
Jul 18 17:27:11.689898 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.689722    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="afec0482-b062-4758-b518-850636637cf1" containerName="istio-proxy"
Jul 18 17:27:11.689898 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.689728    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a38bdb90-a757-4169-8864-d08232ce0efe" containerName="maas-api"
Jul 18 17:27:11.689898 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.689735    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="afec0482-b062-4758-b518-850636637cf1" containerName="istio-proxy"
Jul 18 17:27:11.689898 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.689742    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a6b64bc-f9a8-4f2b-bf6e-49b5008fa6f1" containerName="revoke-keys"
Jul 18 17:27:11.689898 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.689793    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afec0482-b062-4758-b518-850636637cf1" containerName="istio-proxy"
Jul 18 17:27:11.689898 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.689824    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="afec0482-b062-4758-b518-850636637cf1" containerName="istio-proxy"
Jul 18 17:27:11.692928 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.692912    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fc66r/must-gather-hj8lj"
Jul 18 17:27:11.696684 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.696664    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fc66r\"/\"kube-root-ca.crt\""
Jul 18 17:27:11.696801 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.696772    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fc66r\"/\"openshift-service-ca.crt\""
Jul 18 17:27:11.697890 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.697871    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fc66r\"/\"default-dockercfg-sttkp\""
Jul 18 17:27:11.698952 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.698921    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:27:11.703651 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.703630    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fc66r/must-gather-hj8lj"]
Jul 18 17:27:11.762578 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.762544    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2ld9\" (UniqueName: \"kubernetes.io/projected/34cafca2-3acb-481b-9d24-998055c73614-kube-api-access-g2ld9\") pod \"must-gather-hj8lj\" (UID: \"34cafca2-3acb-481b-9d24-998055c73614\") " pod="openshift-must-gather-fc66r/must-gather-hj8lj"
Jul 18 17:27:11.762726 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.762605    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/34cafca2-3acb-481b-9d24-998055c73614-must-gather-output\") pod \"must-gather-hj8lj\" (UID: \"34cafca2-3acb-481b-9d24-998055c73614\") " pod="openshift-must-gather-fc66r/must-gather-hj8lj"
Jul 18 17:27:11.770708 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.770681    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:27:11.863633 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.863592    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ld9\" (UniqueName: \"kubernetes.io/projected/34cafca2-3acb-481b-9d24-998055c73614-kube-api-access-g2ld9\") pod \"must-gather-hj8lj\" (UID: \"34cafca2-3acb-481b-9d24-998055c73614\") " pod="openshift-must-gather-fc66r/must-gather-hj8lj"
Jul 18 17:27:11.863801 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.863644    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/34cafca2-3acb-481b-9d24-998055c73614-must-gather-output\") pod \"must-gather-hj8lj\" (UID: \"34cafca2-3acb-481b-9d24-998055c73614\") " pod="openshift-must-gather-fc66r/must-gather-hj8lj"
Jul 18 17:27:11.864055 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.864033    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/34cafca2-3acb-481b-9d24-998055c73614-must-gather-output\") pod \"must-gather-hj8lj\" (UID: \"34cafca2-3acb-481b-9d24-998055c73614\") " pod="openshift-must-gather-fc66r/must-gather-hj8lj"
Jul 18 17:27:11.872320 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:11.872299    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2ld9\" (UniqueName: \"kubernetes.io/projected/34cafca2-3acb-481b-9d24-998055c73614-kube-api-access-g2ld9\") pod \"must-gather-hj8lj\" (UID: \"34cafca2-3acb-481b-9d24-998055c73614\") " pod="openshift-must-gather-fc66r/must-gather-hj8lj"
Jul 18 17:27:12.002349 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:12.002261    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fc66r/must-gather-hj8lj"
Jul 18 17:27:12.134542 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:12.134516    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fc66r/must-gather-hj8lj"]
Jul 18 17:27:12.136795 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:27:12.136770    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34cafca2_3acb_481b_9d24_998055c73614.slice/crio-0707d295781f16fb7bf401b2b69035da07030f9de9a707511ee514941ea564eb WatchSource:0}: Error finding container 0707d295781f16fb7bf401b2b69035da07030f9de9a707511ee514941ea564eb: Status 404 returned error can't find the container with id 0707d295781f16fb7bf401b2b69035da07030f9de9a707511ee514941ea564eb
Jul 18 17:27:12.138717 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:12.138700    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 18 17:27:13.026866 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:13.026762    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fc66r/must-gather-hj8lj" event={"ID":"34cafca2-3acb-481b-9d24-998055c73614","Type":"ContainerStarted","Data":"0707d295781f16fb7bf401b2b69035da07030f9de9a707511ee514941ea564eb"}
Jul 18 17:27:14.035713 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:14.035667    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fc66r/must-gather-hj8lj" event={"ID":"34cafca2-3acb-481b-9d24-998055c73614","Type":"ContainerStarted","Data":"1dfd12b55773063b608e0da24ba234905499d7bbf653ac9ec30fd72481826b59"}
Jul 18 17:27:14.036253 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:14.035727    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fc66r/must-gather-hj8lj" event={"ID":"34cafca2-3acb-481b-9d24-998055c73614","Type":"ContainerStarted","Data":"214fa2d985e1541366451d0785b998f0f5282cd9ff4d82d60acc23881e840414"}
Jul 18 17:27:14.053719 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:14.053663    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fc66r/must-gather-hj8lj" podStartSLOduration=1.991553277 podStartE2EDuration="3.053642573s" totalImagesPullingTime="1.062089296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 17:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:27:14.050579811 +0000 UTC m=+3052.748410498" watchObservedRunningTime="2026-07-18 17:27:14.053642573 +0000 UTC m=+3052.751473258"
Jul 18 17:27:14.798364 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:14.798299    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wzt7c_5a99a42b-8462-4cc8-9512-e9316a46672f/global-pull-secret-syncer/0.log"
Jul 18 17:27:14.848972 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:14.848931    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log"
Jul 18 17:27:14.991826 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:14.991783    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-77.ec2.internal_a00913a0b98636dd43acae671203e933/haproxy/0.log"
Jul 18 17:27:20.631605 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:20.631565    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d86c54866-njv76_92cf2d68-2200-4256-a36b-0465b913a01f/authorino/0.log"
Jul 18 17:27:20.665868 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:20.664965    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-9sqhw_53346a72-5b96-4904-8537-e16d68b3d7f6/manager/0.log"
Jul 18 17:27:23.183739 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:23.183699    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-p8242_f26b4eeb-fd35-4a94-beba-9b08799df449/monitoring-plugin/0.log"
Jul 18 17:27:23.298300 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:23.298264    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mwgzb_57df3c7d-e80d-49b0-b1d0-4d21cbfc519f/node-exporter/0.log"
Jul 18 17:27:23.321405 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:23.321375    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mwgzb_57df3c7d-e80d-49b0-b1d0-4d21cbfc519f/kube-rbac-proxy/0.log"
Jul 18 17:27:23.344216 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:23.344184    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mwgzb_57df3c7d-e80d-49b0-b1d0-4d21cbfc519f/init-textfile/0.log"
Jul 18 17:27:23.531491 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:23.531369    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51e008b8-832b-4103-8d3a-8ecffd21ecc7/prometheus/0.log"
Jul 18 17:27:23.554046 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:23.554009    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51e008b8-832b-4103-8d3a-8ecffd21ecc7/config-reloader/0.log"
Jul 18 17:27:23.576380 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:23.576353    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51e008b8-832b-4103-8d3a-8ecffd21ecc7/thanos-sidecar/0.log"
Jul 18 17:27:23.603167 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:23.603135    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51e008b8-832b-4103-8d3a-8ecffd21ecc7/kube-rbac-proxy-web/0.log"
Jul 18 17:27:23.630186 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:23.630155    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51e008b8-832b-4103-8d3a-8ecffd21ecc7/kube-rbac-proxy/0.log"
Jul 18 17:27:23.653941 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:23.653898    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51e008b8-832b-4103-8d3a-8ecffd21ecc7/kube-rbac-proxy-thanos/0.log"
Jul 18 17:27:23.675024 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:23.674993    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_51e008b8-832b-4103-8d3a-8ecffd21ecc7/init-config-reloader/0.log"
Jul 18 17:27:23.702738 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:23.702706    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-8pklg_ea4f34b8-1180-4804-923e-c5868ea12113/prometheus-operator/0.log"
Jul 18 17:27:23.720723 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:23.720691    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-8pklg_ea4f34b8-1180-4804-923e-c5868ea12113/kube-rbac-proxy/0.log"
Jul 18 17:27:23.745628 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:23.745600    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-xmhf6_9bdcba4c-e0df-485e-a4be-05d82d176e0a/prometheus-operator-admission-webhook/0.log"
Jul 18 17:27:24.893298 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:24.893257    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"]
Jul 18 17:27:24.897395 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:24.897367    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:24.905000 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:24.904956    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"]
Jul 18 17:27:25.000357 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.000309    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b981705d-7ebd-4e3c-b6e9-aebeac2e6719-sys\") pod \"perf-node-gather-daemonset-j9c9p\" (UID: \"b981705d-7ebd-4e3c-b6e9-aebeac2e6719\") " pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.000776 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.000757    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b981705d-7ebd-4e3c-b6e9-aebeac2e6719-lib-modules\") pod \"perf-node-gather-daemonset-j9c9p\" (UID: \"b981705d-7ebd-4e3c-b6e9-aebeac2e6719\") " pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.000966 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.000945    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b981705d-7ebd-4e3c-b6e9-aebeac2e6719-proc\") pod \"perf-node-gather-daemonset-j9c9p\" (UID: \"b981705d-7ebd-4e3c-b6e9-aebeac2e6719\") " pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.001140 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.001112    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22vh7\" (UniqueName: \"kubernetes.io/projected/b981705d-7ebd-4e3c-b6e9-aebeac2e6719-kube-api-access-22vh7\") pod \"perf-node-gather-daemonset-j9c9p\" (UID: \"b981705d-7ebd-4e3c-b6e9-aebeac2e6719\") " pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.001278 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.001262    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b981705d-7ebd-4e3c-b6e9-aebeac2e6719-podres\") pod \"perf-node-gather-daemonset-j9c9p\" (UID: \"b981705d-7ebd-4e3c-b6e9-aebeac2e6719\") " pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.102414 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.102363    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22vh7\" (UniqueName: \"kubernetes.io/projected/b981705d-7ebd-4e3c-b6e9-aebeac2e6719-kube-api-access-22vh7\") pod \"perf-node-gather-daemonset-j9c9p\" (UID: \"b981705d-7ebd-4e3c-b6e9-aebeac2e6719\") " pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.102595 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.102429    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b981705d-7ebd-4e3c-b6e9-aebeac2e6719-podres\") pod \"perf-node-gather-daemonset-j9c9p\" (UID: \"b981705d-7ebd-4e3c-b6e9-aebeac2e6719\") " pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.102595 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.102540    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b981705d-7ebd-4e3c-b6e9-aebeac2e6719-sys\") pod \"perf-node-gather-daemonset-j9c9p\" (UID: \"b981705d-7ebd-4e3c-b6e9-aebeac2e6719\") " pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.102710 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.102617    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b981705d-7ebd-4e3c-b6e9-aebeac2e6719-podres\") pod \"perf-node-gather-daemonset-j9c9p\" (UID: \"b981705d-7ebd-4e3c-b6e9-aebeac2e6719\") " pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.102710 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.102643    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b981705d-7ebd-4e3c-b6e9-aebeac2e6719-sys\") pod \"perf-node-gather-daemonset-j9c9p\" (UID: \"b981705d-7ebd-4e3c-b6e9-aebeac2e6719\") " pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.102790 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.102752    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b981705d-7ebd-4e3c-b6e9-aebeac2e6719-lib-modules\") pod \"perf-node-gather-daemonset-j9c9p\" (UID: \"b981705d-7ebd-4e3c-b6e9-aebeac2e6719\") " pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.102790 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.102778    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b981705d-7ebd-4e3c-b6e9-aebeac2e6719-proc\") pod \"perf-node-gather-daemonset-j9c9p\" (UID: \"b981705d-7ebd-4e3c-b6e9-aebeac2e6719\") " pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.102919 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.102896    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b981705d-7ebd-4e3c-b6e9-aebeac2e6719-proc\") pod \"perf-node-gather-daemonset-j9c9p\" (UID: \"b981705d-7ebd-4e3c-b6e9-aebeac2e6719\") " pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.102973 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.102918    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b981705d-7ebd-4e3c-b6e9-aebeac2e6719-lib-modules\") pod \"perf-node-gather-daemonset-j9c9p\" (UID: \"b981705d-7ebd-4e3c-b6e9-aebeac2e6719\") " pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.110308 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.110282    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22vh7\" (UniqueName: \"kubernetes.io/projected/b981705d-7ebd-4e3c-b6e9-aebeac2e6719-kube-api-access-22vh7\") pod \"perf-node-gather-daemonset-j9c9p\" (UID: \"b981705d-7ebd-4e3c-b6e9-aebeac2e6719\") " pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.210485 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.210398    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:25.413048 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:25.413020    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"]
Jul 18 17:27:25.415126 ip-10-0-142-77 kubenswrapper[2554]: W0718 17:27:25.415091    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb981705d_7ebd_4e3c_b6e9_aebeac2e6719.slice/crio-5bd7c0ecb89bde4ff6f7f8567fc5dcd86d2cb72b1dbc6f319ce25d76f67231fe WatchSource:0}: Error finding container 5bd7c0ecb89bde4ff6f7f8567fc5dcd86d2cb72b1dbc6f319ce25d76f67231fe: Status 404 returned error can't find the container with id 5bd7c0ecb89bde4ff6f7f8567fc5dcd86d2cb72b1dbc6f319ce25d76f67231fe
Jul 18 17:27:26.119836 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:26.119797    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p" event={"ID":"b981705d-7ebd-4e3c-b6e9-aebeac2e6719","Type":"ContainerStarted","Data":"eebe327ccfdcc1f235a3c696a2cd89309dcfc9b3ec305bef6aebff0072269fcd"}
Jul 18 17:27:26.120445 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:26.120420    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p" event={"ID":"b981705d-7ebd-4e3c-b6e9-aebeac2e6719","Type":"ContainerStarted","Data":"5bd7c0ecb89bde4ff6f7f8567fc5dcd86d2cb72b1dbc6f319ce25d76f67231fe"}
Jul 18 17:27:26.120676 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:26.120649    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:26.139347 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:26.139287    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p" podStartSLOduration=2.139268131 podStartE2EDuration="2.139268131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 17:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 17:27:26.137750146 +0000 UTC m=+3064.835580829" watchObservedRunningTime="2026-07-18 17:27:26.139268131 +0000 UTC m=+3064.837098817"
Jul 18 17:27:28.041760 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:28.041728    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-994h4_9bbd0e94-6c81-4eaa-abb9-522600419f63/dns/0.log"
Jul 18 17:27:28.065408 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:28.065363    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-994h4_9bbd0e94-6c81-4eaa-abb9-522600419f63/kube-rbac-proxy/0.log"
Jul 18 17:27:28.178633 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:28.178595    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sz6hk_8c71dd87-f690-4c8e-b286-6eff3854fd93/dns-node-resolver/0.log"
Jul 18 17:27:28.750931 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:28.750901    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7dcbf79b9c-x7qxx_95484bc2-9453-4837-8f68-45a375d918bf/registry/0.log"
Jul 18 17:27:28.798762 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:28.798734    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6jtwc_a48a86ca-3a85-46b8-ada3-0125ec0e9546/node-ca/0.log"
Jul 18 17:27:29.922267 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:29.922231    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c475ff8c-hb9q5_40d62415-b2ff-4eb9-9b5b-ec90c705184b/kube-auth-proxy/0.log"
Jul 18 17:27:30.820935 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:30.820908    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-psptl_09878619-4f85-4acb-bce3-94a409e582d1/serve-healthcheck-canary/0.log"
Jul 18 17:27:31.347032 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:31.346975    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-kvsvt_05af46b3-3c72-416b-b837-d9412910c1b5/insights-operator/0.log"
Jul 18 17:27:31.348236 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:31.348211    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-kvsvt_05af46b3-3c72-416b-b837-d9412910c1b5/insights-operator/1.log"
Jul 18 17:27:31.451686 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:31.451645    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w95dc_0728aabe-3077-4396-98c2-f6f7c1ec84db/kube-rbac-proxy/0.log"
Jul 18 17:27:31.474871 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:31.474821    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w95dc_0728aabe-3077-4396-98c2-f6f7c1ec84db/exporter/0.log"
Jul 18 17:27:31.500882 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:31.500841    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w95dc_0728aabe-3077-4396-98c2-f6f7c1ec84db/extractor/0.log"
Jul 18 17:27:32.137535 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:32.137501    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fc66r/perf-node-gather-daemonset-j9c9p"
Jul 18 17:27:34.542156 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:34.542115    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c768ddb5d-whpnh_7af4d913-b0bd-4af3-92e2-2ae1ad9debf4/manager/0.log"
Jul 18 17:27:34.561536 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:34.561490    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c768ddb5d-whpnh_7af4d913-b0bd-4af3-92e2-2ae1ad9debf4/copy-manifests/0.log"
Jul 18 17:27:35.641684 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:35.641652    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c475ff8c-hb9q5_40d62415-b2ff-4eb9-9b5b-ec90c705184b/kube-auth-proxy/0.log"
Jul 18 17:27:35.927910 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:35.927809    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-867f5b7897-mwfhn_038c1dca-874f-49b8-986c-cbe3e31c6721/manager/0.log"
Jul 18 17:27:37.110004 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:37.109921    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c475ff8c-hb9q5_40d62415-b2ff-4eb9-9b5b-ec90c705184b/kube-auth-proxy/0.log"
Jul 18 17:27:39.927710 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:39.927672    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:27:39.982867 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:39.982816    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kf9nq_f186e77b-ccfd-461c-8cb8-e4f3a746fbeb/konnectivity-agent/0.log" limit=1024
Jul 18 17:27:40.916268 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:40.916242    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-ml29g_5be0e73f-85a3-44af-b6fa-3a385abaf6c4/migrator/0.log"
Jul 18 17:27:40.940757 ip-10-0-142-77 kubenswrapper[2554]: I0718 17:27:40.940731    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-ml29g_5be0e73f-85a3-44af-b6fa-3a385abaf6c4/graceful-termination/0.log"