Jul 09 11:11:56.296073 ip-10-0-138-176 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 11:11:56.683538 ip-10-0-138-176 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 11:11:56.683538 ip-10-0-138-176 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 11:11:56.683538 ip-10-0-138-176 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 11:11:56.683538 ip-10-0-138-176 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 11:11:56.683538 ip-10-0-138-176 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 11:11:56.686593 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.686448    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 11:11:56.690921 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690906    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 11:11:56.690959 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690921    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 11:11:56.690959 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690940    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 11:11:56.690959 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690944    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 11:11:56.690959 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690948    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 11:11:56.690959 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690951    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 11:11:56.690959 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690953    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 11:11:56.690959 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690956    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 11:11:56.690959 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690959    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 11:11:56.690959 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690962    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690965    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690968    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690971    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690973    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690976    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690979    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690981    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690984    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690987    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690989    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.690992    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691005    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691008    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691010    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691013    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691015    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691018    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691021    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691023    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 11:11:56.691179 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691026    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691029    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691031    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691033    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691036    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691038    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691040    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691043    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691045    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691048    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691051    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691053    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691056    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691059    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691063    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691066    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691069    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691071    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691075    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691079    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 11:11:56.691654 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691082    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691084    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691087    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691090    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691092    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691101    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691105    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691108    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691111    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691114    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691117    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691119    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691122    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691124    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691127    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691129    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691132    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691136    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691139    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 11:11:56.692160 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691141    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691145    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691147    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691150    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691153    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691156    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691158    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691164    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691166    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691169    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691171    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691174    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691177    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691179    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691182    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691185    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691187    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691190    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691193    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691203    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 11:11:56.692626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691205    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691208    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691210    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691213    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691215    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691218    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691221    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691223    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691226    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691229    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691231    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691971    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691978    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691981    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691984    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691987    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691990    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691993    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691996    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.691999    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 11:11:56.693129 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692002    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692005    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692007    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692010    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692013    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692016    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692020    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692024    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692027    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692030    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692032    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692035    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692038    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692041    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692043    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692046    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692048    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692051    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692053    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692056    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 11:11:56.693615 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692059    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692061    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692064    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692066    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692069    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692071    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692074    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692076    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692079    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692081    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692085    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692088    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692091    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692094    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692097    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692099    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692102    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692104    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692107    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692109    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 11:11:56.694168 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692111    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692114    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692118    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692120    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692123    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692132    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692135    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692137    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692140    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692143    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692145    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692149    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692152    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692155    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692158    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692161    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692164    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692167    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692170    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692172    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 11:11:56.694697 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692175    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692178    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692181    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692183    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692186    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692189    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692191    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692194    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692196    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692199    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692201    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692204    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692208    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692210    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692213    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692215    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692218    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692220    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692228    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 11:11:56.695200 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692231    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692233    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692237    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692241    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692244    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692246    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692249    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692252    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692254    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692257    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692259    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692737    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692742    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692745    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692748    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692751    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692753    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692756    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692759    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 11:11:56.695652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692761    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692764    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692766    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692769    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692771    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692774    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692776    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692793    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692795    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692798    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692800    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692803    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692812    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692815    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692818    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692820    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692823    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692825    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692828    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 11:11:56.696132 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692830    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692833    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692836    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692839    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692841    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692844    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692847    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692850    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692853    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692855    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692858    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692863    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692866    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692869    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692872    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692876    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692878    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692881    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692884    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692887    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 11:11:56.696609 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692890    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692893    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692896    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692899    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692901    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692904    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692912    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692914    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692917    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692919    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692921    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692924    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692927    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692930    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692932    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692934    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692937    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692939    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692942    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692945    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 11:11:56.697104 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692947    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692950    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692952    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692955    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692963    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692965    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692968    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692970    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692973    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692975    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692977    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692980    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692982    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692985    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692987    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692990    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692994    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692997    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.692999    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 11:11:56.697594 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693008    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693011    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693013    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693017    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693020    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693023    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693026    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693029    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693032    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693035    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693037    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693041    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693043    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693115    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693121    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693136    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693144    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693149    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693154    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693158    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693162    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 11:11:56.698075 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693165    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693168    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693172    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693175    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693178    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693181    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693184    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693187    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693190    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693192    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693199    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693202    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693211    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693214    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693217    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693221    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693224    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693227    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693231    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693234    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693237    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693240    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693247    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693250    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 11:11:56.698586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693254    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693257    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693260    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693263    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693266    2563 flags.go:64] FLAG: --enable-server="true"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693269    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693276    2563 flags.go:64] FLAG: --event-burst="100"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693279    2563 flags.go:64] FLAG: --event-qps="50"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693282    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693285    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693288    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693292    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693294    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693297    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693300    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693303    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693306    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693309    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693311    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693314    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693317    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693325    2563 flags.go:64] FLAG: --feature-gates=""
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693329    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693332    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693335    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 11:11:56.699205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693339    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693342    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693344    2563 flags.go:64] FLAG: --help="false"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693347    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-138-176.ec2.internal"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693350    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693354    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693358    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693361    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693364    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693367    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693370    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693373    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693375    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693379    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693382    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693385    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693388    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693390    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693393    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693396    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693399    2563 flags.go:64] FLAG: --lock-file=""
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693402    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693404    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693407    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 11:11:56.700164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693413    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693416    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693418    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693421    2563 flags.go:64] FLAG: --logging-format="text"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693424    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693433    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693436    2563 flags.go:64] FLAG: --manifest-url=""
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693440    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693445    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693448    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693451    2563 flags.go:64] FLAG: --max-pods="110"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693454    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693457    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693462    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693465    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693468    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693470    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693473    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693480    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693483    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693486    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693489    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693492    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 09 11:11:56.700842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693497    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693499    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693503    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693506    2563 flags.go:64] FLAG: --port="10250"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693509    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693511    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-013944061dd8b8ede"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693515    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693517    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693520    2563 flags.go:64] FLAG: --register-node="true"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693523    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693526    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693529    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693532    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693534    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693538    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693547    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693551    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693554    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693557    2563 flags.go:64] FLAG: --runonce="false"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693560    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693563    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693565    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693568    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693572    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693575    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693578    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 11:11:56.701419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693581    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693584    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693587    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693590    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693593    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693596    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693599    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693601    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693606    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693609    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693612    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693618    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693621    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693624    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693627    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693630    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693632    2563 flags.go:64] FLAG: --v="2"
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693636    2563 flags.go:64] FLAG: --version="false"
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693640    2563 flags.go:64] FLAG: --vmodule=""
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693644    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.693647    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693777    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693796    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693800    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 11:11:56.702062 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693803    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693807    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693811    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693814    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693817    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693822    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693825    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693828    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693831    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693833    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693836    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693839    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693841    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693844    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693846    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693849    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693852    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693854    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693857    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 11:11:56.702652 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693861    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693864    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693867    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693870    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693873    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693876    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693879    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693881    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693884    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693887    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693889    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693893    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693896    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693899    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693901    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693904    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693906    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693909    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693914    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693917    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 11:11:56.703146 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693919    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693922    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693924    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693927    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693930    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693932    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693936    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693939    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693942    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693944    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693947    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693950    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693952    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693955    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693957    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693960    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693962    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693965    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693967    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693970    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 11:11:56.703638 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693973    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693975    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693978    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693982    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693985    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693987    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693990    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693993    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693995    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.693998    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694002    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694004    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694007    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694009    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694012    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694015    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694017    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694020    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694022    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694025    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 11:11:56.704174 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694027    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694030    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694033    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694035    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694038    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694040    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694043    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694046    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694048    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694050    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694053    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694056    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694058    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694060    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694063    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694067    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 11:11:56.704674 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.694069    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 11:11:56.705081 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.694075    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 11:11:56.705081 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.702233    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 09 11:11:56.705081 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.702249    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 11:11:56.705081 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.702269    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 11:11:56.705081 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.702275    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 11:11:56.705081 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702333    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 11:11:56.705081 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702338    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 11:11:56.705081 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702341    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 11:11:56.705081 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702345    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 11:11:56.705081 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702348    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 11:11:56.705081 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702351    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 11:11:56.705081 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702355    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 11:11:56.705081 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702359    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 11:11:56.705081 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702363    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 11:11:56.705081 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702366    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702369    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702371    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702374    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702377    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702379    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702382    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702384    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702387    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702390    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702392    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702395    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702398    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702401    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702403    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702406    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702409    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702411    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702415    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702417    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 11:11:56.705445 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702421    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702423    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702426    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702429    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702431    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702434    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702436    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702439    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702441    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702444    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702447    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702449    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702452    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702455    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702457    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702460    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702462    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702465    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702468    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702471    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 11:11:56.705982 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702473    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702475    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702478    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702480    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702483    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702485    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702488    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702490    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702494    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702497    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702500    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702503    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702507    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702510    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702512    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702515    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702518    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702520    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702523    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702526    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 11:11:56.706479 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702528    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702530    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702533    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702536    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702538    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702541    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702543    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702546    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702548    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702550    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702554    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702557    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702560    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702562    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702565    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702567    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702571    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702574    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702578    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 11:11:56.706973 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702581    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 11:11:56.707456 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702596    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 11:11:56.707456 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702599    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 11:11:56.707456 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702602    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 11:11:56.707456 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702605    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 11:11:56.707456 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702608    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 11:11:56.707456 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702611    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 11:11:56.707456 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702613    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 11:11:56.707456 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702616    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 11:11:56.707456 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702618    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 11:11:56.707456 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702621    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 11:11:56.707456 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.702626    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 11:11:56.707456 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702736    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 11:11:56.707456 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702741    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 11:11:56.707456 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702744    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702747    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702750    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702753    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702755    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702758    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702760    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702763    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702765    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702768    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702771    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702773    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702776    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702794    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702797    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702800    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702803    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702806    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702809    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 11:11:56.707819 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702811    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702814    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702817    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702819    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702822    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702824    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702827    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702829    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702832    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702836    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702841    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702843    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702846    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702849    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702852    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702855    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702858    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702860    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702863    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702866    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 11:11:56.708287 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702869    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702871    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702874    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702877    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702880    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702883    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702886    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702888    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702891    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702893    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702896    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702898    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702900    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702903    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702905    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702908    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702910    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702913    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702915    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702918    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 11:11:56.708774 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702920    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702923    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702925    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702928    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702932    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702935    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702938    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702941    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702943    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702946    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702948    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702951    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702954    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702956    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702959    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702961    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702964    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702967    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702970    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702972    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 11:11:56.709278 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702975    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702977    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702980    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702982    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702984    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702987    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702990    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702994    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702997    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.702999    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.703002    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.703004    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.703007    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.703009    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.703012    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.703014    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.703017    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 11:11:56.709760 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:56.703020    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 11:11:56.710195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.703024    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 11:11:56.710195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.703151    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 11:11:56.710195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.705360    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 11:11:56.710195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.706604    2563 server.go:1019] "Starting client certificate rotation"
Jul 09 11:11:56.710195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.706702    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 11:11:56.710195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.706740    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 11:11:56.710195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.708798    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 11:11:56.710195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.709910    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 11:11:56.720684 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.720664    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 09 11:11:56.724493 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.724468    2563 log.go:25] "Validated CRI v1 image API"
Jul 09 11:11:56.726203 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.726091    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 11:11:56.732616 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.732595    2563 fs.go:135] Filesystem UUIDs: map[314d9026-6054-4602-afc6-267490801b85:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 bbf6fb44-d32f-45ea-9141-7386bdb5fc21:/dev/nvme0n1p4]
Jul 09 11:11:56.732616 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.732614    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 11:11:56.737590 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.737486    2563 manager.go:216] Machine: {Timestamp:2026-07-09 11:11:56.737007039 +0000 UTC m=+0.328738194 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099289 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a96d157f7603c2d932e1bc8cb6abe SystemUUID:ec2a96d1-57f7-603c-2d93-2e1bc8cb6abe BootID:90d1b2cf-51ac-4799-a389-5447c99c4136 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bb:f1:90:e0:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bb:f1:90:e0:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:38:e7:34:2a:34 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 11:11:56.737590 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.737588    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 11:11:56.737701 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.737664    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 11:11:56.738761 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.738731    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 11:11:56.738902 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.738765    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-176.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 11:11:56.739425 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.739416    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 11:11:56.739458 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.739427    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 11:11:56.739458 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.739435    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 11:11:56.739995 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.739985    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 11:11:56.741440 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.741430    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 11:11:56.744578 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.744567    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 11:11:56.744692 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.744683    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 11:11:56.745307 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.745299    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 11:11:56.745339 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.745312    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 11:11:56.745339 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.745325    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 11:11:56.745339 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.745334    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 09 11:11:56.745419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.745343    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 11:11:56.745681 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.745668    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 11:11:56.745740 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.745692    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 11:11:56.747725 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.747704    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 09 11:11:56.749662 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.749649    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 11:11:56.749722 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.749678    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 11:11:56.750689 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.750678    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 11:11:56.750733 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.750694    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 11:11:56.750733 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.750702    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 11:11:56.750733 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.750707    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 11:11:56.750733 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.750715    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 11:11:56.750733 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.750721    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 11:11:56.750733 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.750728    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 11:11:56.750733 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.750734    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 11:11:56.750972 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.750740    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 11:11:56.750972 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.750745    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 11:11:56.750972 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.750753    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 11:11:56.750972 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.750796    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 11:11:56.750972 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.750828    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 11:11:56.750972 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.750836    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 11:11:56.754137 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.754123    2563 server.go:1268] "Started kubelet"
Jul 09 11:11:56.754310 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.754259    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 11:11:56.754406 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.754329    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 11:11:56.754838 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.754810    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 11:11:56.754945 ip-10-0-138-176 systemd[1]: Started Kubernetes Kubelet.
Jul 09 11:11:56.755933 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.755918    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 11:11:56.757213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.757197    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 11:11:56.760397 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.760379    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 11:11:56.762560 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.762541    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 11:11:56.762560 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.762549    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 11:11:56.762757 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.762734    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found"
Jul 09 11:11:56.763181 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.763159    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 11:11:56.763269 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.763163    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 11:11:56.763269 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.763196    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 11:11:56.763369 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.763283    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 11:11:56.763369 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.763293    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 11:11:56.763490 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.763374    2563 factory.go:55] Registering systemd factory
Jul 09 11:11:56.763490 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.763387    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 09 11:11:56.763751 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.763732    2563 factory.go:153] Registering CRI-O factory
Jul 09 11:11:56.763751 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.763753    2563 factory.go:223] Registration of the crio container factory successfully
Jul 09 11:11:56.763893 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.763770    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 11:11:56.763893 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.763829    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 11:11:56.763893 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.763852    2563 factory.go:103] Registering Raw factory
Jul 09 11:11:56.763893 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.763876    2563 manager.go:1196] Started watching for new ooms in manager
Jul 09 11:11:56.764039 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.764002    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 11:11:56.764578 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.764560    2563 manager.go:349] Starting recovery of all containers
Jul 09 11:11:56.765210 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.764331    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-176.ec2.internal.18c09aac2f1b5cce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-176.ec2.internal,UID:ip-10-0-138-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-176.ec2.internal,},FirstTimestamp:2026-07-09 11:11:56.754091214 +0000 UTC m=+0.345822371,LastTimestamp:2026-07-09 11:11:56.754091214 +0000 UTC m=+0.345822371,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-176.ec2.internal,}"
Jul 09 11:11:56.765510 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.765480    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-176.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 11:11:56.765629 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.765609    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-176.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 11:11:56.767437 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.767415    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 11:11:56.767616 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.767587    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-176.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 11:11:56.775016 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.774875    2563 manager.go:354] Recovery completed
Jul 09 11:11:56.779293 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.779278    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 11:11:56.781041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.781022    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 11:11:56.781101 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.781053    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 11:11:56.781101 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.781065    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientPID"
Jul 09 11:11:56.781587 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.781574    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 11:11:56.781587 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.781585    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 11:11:56.781673 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.781600    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 11:11:56.781673 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.781575    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 11:11:56.782724 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.782703    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 11:11:56.782831 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.782737    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 11:11:56.782831 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.782760    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientPID"
Jul 09 11:11:56.783644 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.783550    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-176.ec2.internal.18c09aac30b68f9a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-176.ec2.internal,UID:ip-10-0-138-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-176.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-176.ec2.internal,},FirstTimestamp:2026-07-09 11:11:56.781039514 +0000 UTC m=+0.372770668,LastTimestamp:2026-07-09 11:11:56.781039514 +0000 UTC m=+0.372770668,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-176.ec2.internal,}"
Jul 09 11:11:56.784460 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.784443    2563 policy_none.go:49] "None policy: Start"
Jul 09 11:11:56.784522 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.784463    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 11:11:56.784522 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.784474    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 11:11:56.785430 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.785421    2563 policy_none.go:47] "Start"
Jul 09 11:11:56.795416 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.795348    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-176.ec2.internal.18c09aac30b6d64d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-176.ec2.internal,UID:ip-10-0-138-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-176.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-176.ec2.internal,},FirstTimestamp:2026-07-09 11:11:56.781057613 +0000 UTC m=+0.372788768,LastTimestamp:2026-07-09 11:11:56.781057613 +0000 UTC m=+0.372788768,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-176.ec2.internal,}"
Jul 09 11:11:56.804681 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.804582    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-176.ec2.internal.18c09aac30b70156  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-176.ec2.internal,UID:ip-10-0-138-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-176.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-176.ec2.internal,},FirstTimestamp:2026-07-09 11:11:56.78106863 +0000 UTC m=+0.372799785,LastTimestamp:2026-07-09 11:11:56.78106863 +0000 UTC m=+0.372799785,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-176.ec2.internal,}"
Jul 09 11:11:56.811858 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.811777    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-176.ec2.internal.18c09aac30b68f9a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-176.ec2.internal.18c09aac30b68f9a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-176.ec2.internal,UID:ip-10-0-138-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-176.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-176.ec2.internal,},FirstTimestamp:2026-07-09 11:11:56.781039514 +0000 UTC m=+0.372770668,LastTimestamp:2026-07-09 11:11:56.782724128 +0000 UTC m=+0.374455284,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-176.ec2.internal,}"
Jul 09 11:11:56.821372 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.821354    2563 manager.go:337] "Starting Device Plugin manager"
Jul 09 11:11:56.821485 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.821414    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 11:11:56.821485 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.821428    2563 server.go:87] "Starting device plugin registration server"
Jul 09 11:11:56.821713 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.821700    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 11:11:56.821810 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.821715    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 11:11:56.821857 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.821805    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 11:11:56.821928 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.821904    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 11:11:56.821928 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.821918    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 11:11:56.822576 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.822550    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 11:11:56.822655 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.822615    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-176.ec2.internal\" not found"
Jul 09 11:11:56.833028 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.832960    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-176.ec2.internal.18c09aac30b6d64d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-176.ec2.internal.18c09aac30b6d64d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-176.ec2.internal,UID:ip-10-0-138-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-176.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-176.ec2.internal,},FirstTimestamp:2026-07-09 11:11:56.781057613 +0000 UTC m=+0.372788768,LastTimestamp:2026-07-09 11:11:56.782744382 +0000 UTC m=+0.374475539,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-176.ec2.internal,}"
Jul 09 11:11:56.854012 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.853928    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-176.ec2.internal.18c09aac30b70156\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-176.ec2.internal.18c09aac30b70156  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-176.ec2.internal,UID:ip-10-0-138-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-176.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-176.ec2.internal,},FirstTimestamp:2026-07-09 11:11:56.78106863 +0000 UTC m=+0.372799785,LastTimestamp:2026-07-09 11:11:56.782765675 +0000 UTC m=+0.374496830,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-176.ec2.internal,}"
Jul 09 11:11:56.868771 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.868691    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-176.ec2.internal.18c09aac333f9a1c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-176.ec2.internal,UID:ip-10-0-138-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-138-176.ec2.internal,},FirstTimestamp:2026-07-09 11:11:56.823575068 +0000 UTC m=+0.415306213,LastTimestamp:2026-07-09 11:11:56.823575068 +0000 UTC m=+0.415306213,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-176.ec2.internal,}"
Jul 09 11:11:56.869959 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.869936    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5ht8f"
Jul 09 11:11:56.882715 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.882695    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5ht8f"
Jul 09 11:11:56.922320 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.922292    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 11:11:56.923314 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.923295    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 11:11:56.923396 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.923327    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 11:11:56.923396 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.923339    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientPID"
Jul 09 11:11:56.923396 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.923369    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-176.ec2.internal"
Jul 09 11:11:56.942981 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.942901    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 11:11:56.944136 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.944122    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 11:11:56.944226 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.944164    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 11:11:56.944226 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.944193    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 11:11:56.944318 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.944236    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 11:11:56.948178 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.948161    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-176.ec2.internal"
Jul 09 11:11:56.948260 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:56.948186    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-176.ec2.internal\": node \"ip-10-0-138-176.ec2.internal\" not found"
Jul 09 11:11:56.950290 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:56.950278    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 11:11:57.040086 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.040058    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found"
Jul 09 11:11:57.045125 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.045104    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal"]
Jul 09 11:11:57.045181 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.045174    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 11:11:57.046147 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.046123    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 11:11:57.046255 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.046155    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 11:11:57.046255 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.046169    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientPID"
Jul 09 11:11:57.047463 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.047447    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 11:11:57.047637 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.047621    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.047681 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.047652    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 11:11:57.048173 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.048160    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 11:11:57.048250 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.048186    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 11:11:57.048250 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.048200    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientPID"
Jul 09 11:11:57.048250 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.048224    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 11:11:57.048250 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.048246    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 11:11:57.048484 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.048260    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientPID"
Jul 09 11:11:57.049446 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.049431    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.049524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.049464    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 11:11:57.050202 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.050186    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 11:11:57.050285 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.050215    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 11:11:57.050285 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.050228    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeHasSufficientPID"
Jul 09 11:11:57.065656 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.065622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4aca4cc90e184685e6cdf76e3dd114f1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-176.ec2.internal\" (UID: \"4aca4cc90e184685e6cdf76e3dd114f1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.065656 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.065660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2aa1cbda48d30d26959a84b1143a3d09-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"2aa1cbda48d30d26959a84b1143a3d09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.065765 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.065681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2aa1cbda48d30d26959a84b1143a3d09-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"2aa1cbda48d30d26959a84b1143a3d09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.082366 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.082346    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-176.ec2.internal\" not found" node="ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.086741 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.086724    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-176.ec2.internal\" not found" node="ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.140550 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.140523    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found"
Jul 09 11:11:57.166068 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.166043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4aca4cc90e184685e6cdf76e3dd114f1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-176.ec2.internal\" (UID: \"4aca4cc90e184685e6cdf76e3dd114f1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.166213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.166076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2aa1cbda48d30d26959a84b1143a3d09-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"2aa1cbda48d30d26959a84b1143a3d09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.166213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.166095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2aa1cbda48d30d26959a84b1143a3d09-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"2aa1cbda48d30d26959a84b1143a3d09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.166213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.166140    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2aa1cbda48d30d26959a84b1143a3d09-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"2aa1cbda48d30d26959a84b1143a3d09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.166213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.166151    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4aca4cc90e184685e6cdf76e3dd114f1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-176.ec2.internal\" (UID: \"4aca4cc90e184685e6cdf76e3dd114f1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.166213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.166161    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2aa1cbda48d30d26959a84b1143a3d09-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal\" (UID: \"2aa1cbda48d30d26959a84b1143a3d09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.240974 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.240895    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found"
Jul 09 11:11:57.341537 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.341494    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found"
Jul 09 11:11:57.385958 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.385925    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.391320 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.391303    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.442008 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.441976    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found"
Jul 09 11:11:57.542616 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.542534    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-176.ec2.internal\" not found"
Jul 09 11:11:57.574229 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.574196    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 11:11:57.663486 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.663459    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.674536 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.674519    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 11:11:57.675976 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.675964    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal"
Jul 09 11:11:57.689158 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.689133    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 11:11:57.705771 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.705749    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 11:11:57.705916 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.705897    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 11:11:57.705959 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.705922    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 11:11:57.746370 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.746344    2563 apiserver.go:52] "Watching apiserver"
Jul 09 11:11:57.762955 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.762932    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 11:11:57.763146 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.763130    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 11:11:57.763543 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.763524    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-8x75w","kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97","openshift-dns/node-resolver-k7d7z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal","openshift-multus/multus-additional-cni-plugins-8rzgk","openshift-network-diagnostics/network-check-target-bzxtz","openshift-network-operator/iptables-alerter-5f2bh","kube-system/global-pull-secret-syncer-q7c8l","kube-system/konnectivity-agent-2dkxw","openshift-cluster-node-tuning-operator/tuned-wbjqk","openshift-image-registry/node-ca-tx572","openshift-multus/multus-rcn84","openshift-multus/network-metrics-daemon-hphjc"]
Jul 09 11:11:57.765061 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.765045    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.766235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.766219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:11:57.766308 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.766279    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:11:57.767282 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.767269    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:11:57.767337 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.767311    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzxtz" podUID="e3650c56-8363-4f9a-94ec-e5de7bbea057"
Jul 09 11:11:57.767411 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.767395    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 11:11:57.767465 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.767443    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 11:11:57.767718 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.767703    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 11:11:57.767944 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.767923    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m8g8j\""
Jul 09 11:11:57.768387 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.768373    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k7d7z"
Jul 09 11:11:57.769705 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.769691    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.771329 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.771310    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.771616 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.771435    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 11:11:57.771616 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.771488    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 11:11:57.771616 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.771586    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zd7lq\""
Jul 09 11:11:57.772340 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.772321    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 11:11:57.772340 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.772338    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 11:11:57.772488 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.772325    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 11:11:57.772488 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.772338    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tvwpd\""
Jul 09 11:11:57.772488 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.772326    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 11:11:57.772620 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.772568    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5f2bh"
Jul 09 11:11:57.772943 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.772927    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 11:11:57.773856 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.773842    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:11:57.773927 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.773890    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:11:57.775047 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.775033    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2dkxw"
Jul 09 11:11:57.775115 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.775076    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 11:11:57.775429 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.775409    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 11:11:57.775489 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.775460    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r8zpx\""
Jul 09 11:11:57.775531 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.775490    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 11:11:57.775531 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.775421    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 11:11:57.775626 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.775546    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 11:11:57.776448 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.776289    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.776448 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.776301    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 11:11:57.776448 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.776307    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 11:11:57.776448 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.776331    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 11:11:57.776448 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.776366    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r86ns\""
Jul 09 11:11:57.776448 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.776311    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 11:11:57.777850 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.777837    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tx572"
Jul 09 11:11:57.778791 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.778766    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qqhzv\""
Jul 09 11:11:57.779149 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.779127    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.779557 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.779545    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 11:11:57.781021 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.781005    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 11:11:57.781119 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.781022    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 11:11:57.781119 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.781080    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 11:11:57.781119 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.781023    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 11:11:57.781265 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.781196    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 11:11:57.781265 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.781198    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 11:11:57.781440 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.781426    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 11:11:57.781490 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.781429    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 11:11:57.781572 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.781556    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ncqts\""
Jul 09 11:11:57.781642 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.781628    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nrtnk\""
Jul 09 11:11:57.782099 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.782084    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cvfzt\""
Jul 09 11:11:57.834887 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.834868    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qkt87"
Jul 09 11:11:57.858271 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.858238    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qkt87"
Jul 09 11:11:57.864613 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.864596    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 11:11:57.869141 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/366bda55-c1aa-460a-aa23-ca69ab6852af-tmp\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.869235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7hdd\" (UniqueName: \"kubernetes.io/projected/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-kube-api-access-r7hdd\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.869235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1907393-916d-43d8-8ec1-0cb8029ad259-system-cni-dir\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.869235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1907393-916d-43d8-8ec1-0cb8029ad259-cnibin\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.869235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-cni-netd\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.869357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869237    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-cnibin\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.869357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-multus-socket-dir-parent\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.869357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-var-lib-cni-bin\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.869357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-multus-conf-dir\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.869357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/346f1026-54ad-4ae6-9134-e2d1a31e565f-kubelet-config\") pod \"global-pull-secret-syncer-q7c8l\" (UID: \"346f1026-54ad-4ae6-9134-e2d1a31e565f\") " pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:11:57.869357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f1907393-916d-43d8-8ec1-0cb8029ad259-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.869357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-node-log\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.869550 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.869550 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869394    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9-iptables-alerter-script\") pod \"iptables-alerter-5f2bh\" (UID: \"2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9\") " pod="openshift-network-operator/iptables-alerter-5f2bh"
Jul 09 11:11:57.869550 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0aee5e83-b717-46bf-b4d8-07a090cde2c3-konnectivity-ca\") pod \"konnectivity-agent-2dkxw\" (UID: \"0aee5e83-b717-46bf-b4d8-07a090cde2c3\") " pod="kube-system/konnectivity-agent-2dkxw"
Jul 09 11:11:57.869550 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-env-overrides\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.869550 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-sysconfig\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.869550 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldmxc\" (UniqueName: \"kubernetes.io/projected/366bda55-c1aa-460a-aa23-ca69ab6852af-kube-api-access-ldmxc\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.869550 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869534    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00d1d258-b95a-4f9e-88ac-ac35af980a14-serviceca\") pod \"node-ca-tx572\" (UID: \"00d1d258-b95a-4f9e-88ac-ac35af980a14\") " pod="openshift-image-registry/node-ca-tx572"
Jul 09 11:11:57.869741 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869554    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-os-release\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.869741 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-device-dir\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.869741 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-run\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.869741 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-sys\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.869741 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-tuned\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.869741 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-run-netns\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.869741 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2trc\" (UniqueName: \"kubernetes.io/projected/2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9-kube-api-access-t2trc\") pod \"iptables-alerter-5f2bh\" (UID: \"2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9\") " pod="openshift-network-operator/iptables-alerter-5f2bh"
Jul 09 11:11:57.869741 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-cni-bin\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.869741 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-var-lib-kubelet\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.869741 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85bc7340-89d1-40c5-9469-8fd2cb0cd905-multus-daemon-config\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.870033 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flpxt\" (UniqueName: \"kubernetes.io/projected/314935e3-3e91-4cc8-b5fc-15795b54d57a-kube-api-access-flpxt\") pod \"network-metrics-daemon-hphjc\" (UID: \"314935e3-3e91-4cc8-b5fc-15795b54d57a\") " pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:11:57.870033 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-kubelet\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.870033 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-var-lib-openvswitch\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.870033 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-run-openvswitch\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.870033 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-run-ovn\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.870033 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-sysctl-d\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.870033 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-lib-modules\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.870033 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869918    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-run-k8s-cni-cncf-io\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.870033 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zftvf\" (UniqueName: \"kubernetes.io/projected/72b5c8e9-cd8b-4dde-9850-824fbf0818e7-kube-api-access-zftvf\") pod \"node-resolver-k7d7z\" (UID: \"72b5c8e9-cd8b-4dde-9850-824fbf0818e7\") " pod="openshift-dns/node-resolver-k7d7z"
Jul 09 11:11:57.870033 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869971    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9-host-slash\") pod \"iptables-alerter-5f2bh\" (UID: \"2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9\") " pod="openshift-network-operator/iptables-alerter-5f2bh"
Jul 09 11:11:57.870033 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.869988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-multus-cni-dir\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.870033 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870014    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85bc7340-89d1-40c5-9469-8fd2cb0cd905-cni-binary-copy\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-var-lib-cni-multus\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-hostroot\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72b5c8e9-cd8b-4dde-9850-824fbf0818e7-hosts-file\") pod \"node-resolver-k7d7z\" (UID: \"72b5c8e9-cd8b-4dde-9850-824fbf0818e7\") " pod="openshift-dns/node-resolver-k7d7z"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/346f1026-54ad-4ae6-9134-e2d1a31e565f-dbus\") pod \"global-pull-secret-syncer-q7c8l\" (UID: \"346f1026-54ad-4ae6-9134-e2d1a31e565f\") " pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-etc-openvswitch\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-log-socket\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-ovnkube-config\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-etc-kubernetes\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-etc-selinux\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-var-lib-kubelet\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870225    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-ovnkube-script-lib\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-modprobe-d\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-kubernetes\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-systemd\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.870351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-registration-dir\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-sys-fs\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b5c8e9-cd8b-4dde-9850-824fbf0818e7-tmp-dir\") pod \"node-resolver-k7d7z\" (UID: \"72b5c8e9-cd8b-4dde-9850-824fbf0818e7\") " pod="openshift-dns/node-resolver-k7d7z"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0aee5e83-b717-46bf-b4d8-07a090cde2c3-agent-certs\") pod \"konnectivity-agent-2dkxw\" (UID: \"0aee5e83-b717-46bf-b4d8-07a090cde2c3\") " pod="kube-system/konnectivity-agent-2dkxw"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1907393-916d-43d8-8ec1-0cb8029ad259-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-slash\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-system-cni-dir\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f1907393-916d-43d8-8ec1-0cb8029ad259-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs\") pod \"network-metrics-daemon-hphjc\" (UID: \"314935e3-3e91-4cc8-b5fc-15795b54d57a\") " pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1907393-916d-43d8-8ec1-0cb8029ad259-os-release\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1907393-916d-43d8-8ec1-0cb8029ad259-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r54qh\" (UniqueName: \"kubernetes.io/projected/f1907393-916d-43d8-8ec1-0cb8029ad259-kube-api-access-r54qh\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-run-ovn-kubernetes\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-host\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00d1d258-b95a-4f9e-88ac-ac35af980a14-host\") pod \"node-ca-tx572\" (UID: \"00d1d258-b95a-4f9e-88ac-ac35af980a14\") " pod="openshift-image-registry/node-ca-tx572"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47ljc\" (UniqueName: \"kubernetes.io/projected/00d1d258-b95a-4f9e-88ac-ac35af980a14-kube-api-access-47ljc\") pod \"node-ca-tx572\" (UID: \"00d1d258-b95a-4f9e-88ac-ac35af980a14\") " pod="openshift-image-registry/node-ca-tx572"
Jul 09 11:11:57.870774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wltgk\" (UniqueName: \"kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk\") pod \"network-check-target-bzxtz\" (UID: \"e3650c56-8363-4f9a-94ec-e5de7bbea057\") " pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:11:57.871272 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret\") pod \"global-pull-secret-syncer-q7c8l\" (UID: \"346f1026-54ad-4ae6-9134-e2d1a31e565f\") " pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:11:57.871272 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-systemd-units\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.871272 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-ovn-node-metrics-cert\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.871272 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88cw5\" (UniqueName: \"kubernetes.io/projected/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-kube-api-access-88cw5\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.871272 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-sysctl-conf\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.871272 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-run-multus-certs\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.871272 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfdr7\" (UniqueName: \"kubernetes.io/projected/85bc7340-89d1-40c5-9469-8fd2cb0cd905-kube-api-access-hfdr7\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.871272 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-run-netns\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.871272 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-run-systemd\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.871272 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.870872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-socket-dir\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.885093 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.885047    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 11:06:56 +0000 UTC" deadline="2027-12-13 02:22:00.617379023 +0000 UTC"
Jul 09 11:11:57.885161 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.885093    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12519h10m2.732288497s"
Jul 09 11:11:57.971428 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-ovnkube-config\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.971428 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-etc-kubernetes\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.971649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-etc-selinux\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.971649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-var-lib-kubelet\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.971649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-ovnkube-script-lib\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.971649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-modprobe-d\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.971649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-kubernetes\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.971649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-systemd\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.971649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971540    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-var-lib-kubelet\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.971649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.971649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971580    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-etc-selinux\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.971649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-registration-dir\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.971649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971619    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-kubernetes\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.971649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-sys-fs\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.971649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971626    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-modprobe-d\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.971649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971635    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.971649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b5c8e9-cd8b-4dde-9850-824fbf0818e7-tmp-dir\") pod \"node-resolver-k7d7z\" (UID: \"72b5c8e9-cd8b-4dde-9850-824fbf0818e7\") " pod="openshift-dns/node-resolver-k7d7z"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971666    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-registration-dir\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971677    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-sys-fs\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971657    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-systemd\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.971700    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-etc-kubernetes\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972113    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-ovnkube-script-lib\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0aee5e83-b717-46bf-b4d8-07a090cde2c3-agent-certs\") pod \"konnectivity-agent-2dkxw\" (UID: \"0aee5e83-b717-46bf-b4d8-07a090cde2c3\") " pod="kube-system/konnectivity-agent-2dkxw"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972185    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72b5c8e9-cd8b-4dde-9850-824fbf0818e7-tmp-dir\") pod \"node-resolver-k7d7z\" (UID: \"72b5c8e9-cd8b-4dde-9850-824fbf0818e7\") " pod="openshift-dns/node-resolver-k7d7z"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1907393-916d-43d8-8ec1-0cb8029ad259-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-slash\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972201    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-ovnkube-config\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-system-cni-dir\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f1907393-916d-43d8-8ec1-0cb8029ad259-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs\") pod \"network-metrics-daemon-hphjc\" (UID: \"314935e3-3e91-4cc8-b5fc-15795b54d57a\") " pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1907393-916d-43d8-8ec1-0cb8029ad259-os-release\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972361    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-system-cni-dir\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1907393-916d-43d8-8ec1-0cb8029ad259-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.972399 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972322    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-slash\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r54qh\" (UniqueName: \"kubernetes.io/projected/f1907393-916d-43d8-8ec1-0cb8029ad259-kube-api-access-r54qh\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972415    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f1907393-916d-43d8-8ec1-0cb8029ad259-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-run-ovn-kubernetes\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-host\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.972466    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00d1d258-b95a-4f9e-88ac-ac35af980a14-host\") pod \"node-ca-tx572\" (UID: \"00d1d258-b95a-4f9e-88ac-ac35af980a14\") " pod="openshift-image-registry/node-ca-tx572"
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972491    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f1907393-916d-43d8-8ec1-0cb8029ad259-os-release\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ljc\" (UniqueName: \"kubernetes.io/projected/00d1d258-b95a-4f9e-88ac-ac35af980a14-kube-api-access-47ljc\") pod \"node-ca-tx572\" (UID: \"00d1d258-b95a-4f9e-88ac-ac35af980a14\") " pod="openshift-image-registry/node-ca-tx572"
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.972530    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs podName:314935e3-3e91-4cc8-b5fc-15795b54d57a nodeName:}" failed. No retries permitted until 2026-07-09 11:11:58.472511284 +0000 UTC m=+2.064242440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs") pod "network-metrics-daemon-hphjc" (UID: "314935e3-3e91-4cc8-b5fc-15795b54d57a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972540    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltgk\" (UniqueName: \"kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk\") pod \"network-check-target-bzxtz\" (UID: \"e3650c56-8363-4f9a-94ec-e5de7bbea057\") " pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret\") pod \"global-pull-secret-syncer-q7c8l\" (UID: \"346f1026-54ad-4ae6-9134-e2d1a31e565f\") " pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-systemd-units\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972606    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-ovn-node-metrics-cert\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88cw5\" (UniqueName: \"kubernetes.io/projected/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-kube-api-access-88cw5\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-sysctl-conf\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.973235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-run-multus-certs\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfdr7\" (UniqueName: \"kubernetes.io/projected/85bc7340-89d1-40c5-9469-8fd2cb0cd905-kube-api-access-hfdr7\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-run-netns\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-run-systemd\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-socket-dir\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/366bda55-c1aa-460a-aa23-ca69ab6852af-tmp\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7hdd\" (UniqueName: \"kubernetes.io/projected/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-kube-api-access-r7hdd\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1907393-916d-43d8-8ec1-0cb8029ad259-system-cni-dir\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1907393-916d-43d8-8ec1-0cb8029ad259-cnibin\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972804    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-cni-netd\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-cnibin\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-multus-socket-dir-parent\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-var-lib-cni-bin\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-multus-conf-dir\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972892    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f1907393-916d-43d8-8ec1-0cb8029ad259-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/346f1026-54ad-4ae6-9134-e2d1a31e565f-kubelet-config\") pod \"global-pull-secret-syncer-q7c8l\" (UID: \"346f1026-54ad-4ae6-9134-e2d1a31e565f\") " pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972924    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-host\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.973924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f1907393-916d-43d8-8ec1-0cb8029ad259-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972960    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-run-systemd\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-node-log\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973005    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-run-multus-certs\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.972962    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-run-ovn-kubernetes\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973049    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-run-netns\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973058    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-cni-netd\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9-iptables-alerter-script\") pod \"iptables-alerter-5f2bh\" (UID: \"2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9\") " pod="openshift-network-operator/iptables-alerter-5f2bh"
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.973087    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0aee5e83-b717-46bf-b4d8-07a090cde2c3-konnectivity-ca\") pod \"konnectivity-agent-2dkxw\" (UID: \"0aee5e83-b717-46bf-b4d8-07a090cde2c3\") " pod="kube-system/konnectivity-agent-2dkxw"
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-env-overrides\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.973143    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret podName:346f1026-54ad-4ae6-9134-e2d1a31e565f nodeName:}" failed. No retries permitted until 2026-07-09 11:11:58.473127558 +0000 UTC m=+2.064858706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret") pod "global-pull-secret-syncer-q7c8l" (UID: "346f1026-54ad-4ae6-9134-e2d1a31e565f") : object "kube-system"/"original-pull-secret" not registered
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-sysconfig\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldmxc\" (UniqueName: \"kubernetes.io/projected/366bda55-c1aa-460a-aa23-ca69ab6852af-kube-api-access-ldmxc\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00d1d258-b95a-4f9e-88ac-ac35af980a14-serviceca\") pod \"node-ca-tx572\" (UID: \"00d1d258-b95a-4f9e-88ac-ac35af980a14\") " pod="openshift-image-registry/node-ca-tx572"
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973246    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-multus-socket-dir-parent\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.974524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-os-release\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-device-dir\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973302    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-sysconfig\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-run\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-sys\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973369    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-multus-conf-dir\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-tuned\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-run-netns\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973424    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-sysctl-conf\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2trc\" (UniqueName: \"kubernetes.io/projected/2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9-kube-api-access-t2trc\") pod \"iptables-alerter-5f2bh\" (UID: \"2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9\") " pod="openshift-network-operator/iptables-alerter-5f2bh"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973462    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-var-lib-cni-bin\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-cni-bin\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-var-lib-kubelet\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973502    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-cni-bin\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85bc7340-89d1-40c5-9469-8fd2cb0cd905-multus-daemon-config\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flpxt\" (UniqueName: \"kubernetes.io/projected/314935e3-3e91-4cc8-b5fc-15795b54d57a-kube-api-access-flpxt\") pod \"network-metrics-daemon-hphjc\" (UID: \"314935e3-3e91-4cc8-b5fc-15795b54d57a\") " pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-kubelet\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973586    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9-iptables-alerter-script\") pod \"iptables-alerter-5f2bh\" (UID: \"2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9\") " pod="openshift-network-operator/iptables-alerter-5f2bh"
Jul 09 11:11:57.975263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-var-lib-openvswitch\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973610    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-run-netns\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-run-openvswitch\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973640    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-env-overrides\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-run-ovn\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-sysctl-d\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-lib-modules\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-run-k8s-cni-cncf-io\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftvf\" (UniqueName: \"kubernetes.io/projected/72b5c8e9-cd8b-4dde-9850-824fbf0818e7-kube-api-access-zftvf\") pod \"node-resolver-k7d7z\" (UID: \"72b5c8e9-cd8b-4dde-9850-824fbf0818e7\") " pod="openshift-dns/node-resolver-k7d7z"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9-host-slash\") pod \"iptables-alerter-5f2bh\" (UID: \"2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9\") " pod="openshift-network-operator/iptables-alerter-5f2bh"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-multus-cni-dir\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973854    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f1907393-916d-43d8-8ec1-0cb8029ad259-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85bc7340-89d1-40c5-9469-8fd2cb0cd905-cni-binary-copy\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973902    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-node-log\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-var-lib-cni-multus\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-hostroot\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72b5c8e9-cd8b-4dde-9850-824fbf0818e7-hosts-file\") pod \"node-resolver-k7d7z\" (UID: \"72b5c8e9-cd8b-4dde-9850-824fbf0818e7\") " pod="openshift-dns/node-resolver-k7d7z"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973196    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-cnibin\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.976041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/346f1026-54ad-4ae6-9134-e2d1a31e565f-dbus\") pod \"global-pull-secret-syncer-q7c8l\" (UID: \"346f1026-54ad-4ae6-9134-e2d1a31e565f\") " pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974009    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f1907393-916d-43d8-8ec1-0cb8029ad259-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-etc-openvswitch\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974011    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-socket-dir\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-log-socket\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974050    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-os-release\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973553    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/346f1026-54ad-4ae6-9134-e2d1a31e565f-kubelet-config\") pod \"global-pull-secret-syncer-q7c8l\" (UID: \"346f1026-54ad-4ae6-9134-e2d1a31e565f\") " pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974074    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-var-lib-kubelet\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.973581    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974143    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-var-lib-cni-multus\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974179    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-hostroot\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974208    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-multus-cni-dir\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974212    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/346f1026-54ad-4ae6-9134-e2d1a31e565f-dbus\") pod \"global-pull-secret-syncer-q7c8l\" (UID: \"346f1026-54ad-4ae6-9134-e2d1a31e565f\") " pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974253    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-run-openvswitch\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974295    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-run-ovn\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974309    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00d1d258-b95a-4f9e-88ac-ac35af980a14-host\") pod \"node-ca-tx572\" (UID: \"00d1d258-b95a-4f9e-88ac-ac35af980a14\") " pod="openshift-image-registry/node-ca-tx572"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974344    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-systemd-units\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974387    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-sysctl-d\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.976855 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974544    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-host-kubelet\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974589    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-log-socket\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974888    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f1907393-916d-43d8-8ec1-0cb8029ad259-system-cni-dir\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.975186    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00d1d258-b95a-4f9e-88ac-ac35af980a14-serviceca\") pod \"node-ca-tx572\" (UID: \"00d1d258-b95a-4f9e-88ac-ac35af980a14\") " pod="openshift-image-registry/node-ca-tx572"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.974626    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f1907393-916d-43d8-8ec1-0cb8029ad259-cnibin\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.975589    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/85bc7340-89d1-40c5-9469-8fd2cb0cd905-multus-daemon-config\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.975848    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0aee5e83-b717-46bf-b4d8-07a090cde2c3-konnectivity-ca\") pod \"konnectivity-agent-2dkxw\" (UID: \"0aee5e83-b717-46bf-b4d8-07a090cde2c3\") " pod="kube-system/konnectivity-agent-2dkxw"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.976229    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/366bda55-c1aa-460a-aa23-ca69ab6852af-etc-tuned\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.976450    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-etc-openvswitch\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.976607    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-sys\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.976649    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72b5c8e9-cd8b-4dde-9850-824fbf0818e7-hosts-file\") pod \"node-resolver-k7d7z\" (UID: \"72b5c8e9-cd8b-4dde-9850-824fbf0818e7\") " pod="openshift-dns/node-resolver-k7d7z"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.976890    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-run\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.976920    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/366bda55-c1aa-460a-aa23-ca69ab6852af-lib-modules\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.977214    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/85bc7340-89d1-40c5-9469-8fd2cb0cd905-host-run-k8s-cni-cncf-io\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.977274    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-var-lib-openvswitch\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.977353    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0aee5e83-b717-46bf-b4d8-07a090cde2c3-agent-certs\") pod \"konnectivity-agent-2dkxw\" (UID: \"0aee5e83-b717-46bf-b4d8-07a090cde2c3\") " pod="kube-system/konnectivity-agent-2dkxw"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.977376    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9-host-slash\") pod \"iptables-alerter-5f2bh\" (UID: \"2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9\") " pod="openshift-network-operator/iptables-alerter-5f2bh"
Jul 09 11:11:57.977601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.977437    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-device-dir\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.979074 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.979054    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/366bda55-c1aa-460a-aa23-ca69ab6852af-tmp\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.979765 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.979741    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 11:11:57.980164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.980138    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-ovn-node-metrics-cert\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.980164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.980158    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85bc7340-89d1-40c5-9469-8fd2cb0cd905-cni-binary-copy\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.983719 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.983695    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47ljc\" (UniqueName: \"kubernetes.io/projected/00d1d258-b95a-4f9e-88ac-ac35af980a14-kube-api-access-47ljc\") pod \"node-ca-tx572\" (UID: \"00d1d258-b95a-4f9e-88ac-ac35af980a14\") " pod="openshift-image-registry/node-ca-tx572"
Jul 09 11:11:57.985278 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.985260    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r54qh\" (UniqueName: \"kubernetes.io/projected/f1907393-916d-43d8-8ec1-0cb8029ad259-kube-api-access-r54qh\") pod \"multus-additional-cni-plugins-8rzgk\" (UID: \"f1907393-916d-43d8-8ec1-0cb8029ad259\") " pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:57.986741 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.986720    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 11:11:57.986741 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.986738    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 11:11:57.986741 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.986747    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wltgk for pod openshift-network-diagnostics/network-check-target-bzxtz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 11:11:57.987044 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:57.986809    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk podName:e3650c56-8363-4f9a-94ec-e5de7bbea057 nodeName:}" failed. No retries permitted until 2026-07-09 11:11:58.486792681 +0000 UTC m=+2.078523844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wltgk" (UniqueName: "kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk") pod "network-check-target-bzxtz" (UID: "e3650c56-8363-4f9a-94ec-e5de7bbea057") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 11:11:57.988904 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.988885    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zftvf\" (UniqueName: \"kubernetes.io/projected/72b5c8e9-cd8b-4dde-9850-824fbf0818e7-kube-api-access-zftvf\") pod \"node-resolver-k7d7z\" (UID: \"72b5c8e9-cd8b-4dde-9850-824fbf0818e7\") " pod="openshift-dns/node-resolver-k7d7z"
Jul 09 11:11:57.989213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.989189    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldmxc\" (UniqueName: \"kubernetes.io/projected/366bda55-c1aa-460a-aa23-ca69ab6852af-kube-api-access-ldmxc\") pod \"tuned-wbjqk\" (UID: \"366bda55-c1aa-460a-aa23-ca69ab6852af\") " pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:57.990565 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.990545    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfdr7\" (UniqueName: \"kubernetes.io/projected/85bc7340-89d1-40c5-9469-8fd2cb0cd905-kube-api-access-hfdr7\") pod \"multus-rcn84\" (UID: \"85bc7340-89d1-40c5-9469-8fd2cb0cd905\") " pod="openshift-multus/multus-rcn84"
Jul 09 11:11:57.991531 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.991511    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88cw5\" (UniqueName: \"kubernetes.io/projected/75dd42d9-d05a-44c9-8d18-664ecaf7feeb-kube-api-access-88cw5\") pod \"ovnkube-node-8x75w\" (UID: \"75dd42d9-d05a-44c9-8d18-664ecaf7feeb\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:57.991833 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.991818    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flpxt\" (UniqueName: \"kubernetes.io/projected/314935e3-3e91-4cc8-b5fc-15795b54d57a-kube-api-access-flpxt\") pod \"network-metrics-daemon-hphjc\" (UID: \"314935e3-3e91-4cc8-b5fc-15795b54d57a\") " pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:11:57.991874 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.991838    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7hdd\" (UniqueName: \"kubernetes.io/projected/2a0b9e74-a148-46e7-9cd9-4ffd6acd3863-kube-api-access-r7hdd\") pod \"aws-ebs-csi-driver-node-xvq97\" (UID: \"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:57.992467 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:57.992452    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2trc\" (UniqueName: \"kubernetes.io/projected/2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9-kube-api-access-t2trc\") pod \"iptables-alerter-5f2bh\" (UID: \"2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9\") " pod="openshift-network-operator/iptables-alerter-5f2bh"
Jul 09 11:11:58.101211 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.101174    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97"
Jul 09 11:11:58.103512 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:58.103481    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aca4cc90e184685e6cdf76e3dd114f1.slice/crio-3c53da916f3b33516776d610fafaf8355c9e00bc8155951de279a863c2f6b021 WatchSource:0}: Error finding container 3c53da916f3b33516776d610fafaf8355c9e00bc8155951de279a863c2f6b021: Status 404 returned error can't find the container with id 3c53da916f3b33516776d610fafaf8355c9e00bc8155951de279a863c2f6b021
Jul 09 11:11:58.106309 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:58.106280    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa1cbda48d30d26959a84b1143a3d09.slice/crio-10b6f78a1840900fb537fce2baecaa94c4f79de57cc94c735714057747a955ea WatchSource:0}: Error finding container 10b6f78a1840900fb537fce2baecaa94c4f79de57cc94c735714057747a955ea: Status 404 returned error can't find the container with id 10b6f78a1840900fb537fce2baecaa94c4f79de57cc94c735714057747a955ea
Jul 09 11:11:58.110683 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.110665    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 11:11:58.111271 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:58.111251    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0b9e74_a148_46e7_9cd9_4ffd6acd3863.slice/crio-e9dc58499122223857087140337fe3b7ec36d225b7b1b1d975f50d8a7fe9da93 WatchSource:0}: Error finding container e9dc58499122223857087140337fe3b7ec36d225b7b1b1d975f50d8a7fe9da93: Status 404 returned error can't find the container with id e9dc58499122223857087140337fe3b7ec36d225b7b1b1d975f50d8a7fe9da93
Jul 09 11:11:58.120412 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.120389    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k7d7z"
Jul 09 11:11:58.127276 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.127241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8rzgk"
Jul 09 11:11:58.128209 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:58.128186    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b5c8e9_cd8b_4dde_9850_824fbf0818e7.slice/crio-ce99ee0f0d9493a0848143a2db108d2fa8d30b2807df23393bd760df1cbd54ca WatchSource:0}: Error finding container ce99ee0f0d9493a0848143a2db108d2fa8d30b2807df23393bd760df1cbd54ca: Status 404 returned error can't find the container with id ce99ee0f0d9493a0848143a2db108d2fa8d30b2807df23393bd760df1cbd54ca
Jul 09 11:11:58.135135 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:58.135115    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1907393_916d_43d8_8ec1_0cb8029ad259.slice/crio-db6a6db8bd6e6965f247ea169285599e7a5a3a01dbe95647c3a88db89b3a5901 WatchSource:0}: Error finding container db6a6db8bd6e6965f247ea169285599e7a5a3a01dbe95647c3a88db89b3a5901: Status 404 returned error can't find the container with id db6a6db8bd6e6965f247ea169285599e7a5a3a01dbe95647c3a88db89b3a5901
Jul 09 11:11:58.144210 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.144191    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:11:58.150395 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.150378    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5f2bh"
Jul 09 11:11:58.150977 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:58.150959    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75dd42d9_d05a_44c9_8d18_664ecaf7feeb.slice/crio-095d288235bbcc870436bc0fb7b5e85875a318283b76a9621906db7a93952ef7 WatchSource:0}: Error finding container 095d288235bbcc870436bc0fb7b5e85875a318283b76a9621906db7a93952ef7: Status 404 returned error can't find the container with id 095d288235bbcc870436bc0fb7b5e85875a318283b76a9621906db7a93952ef7
Jul 09 11:11:58.156987 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:58.156965    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5adaeb_23e5_47a8_a3a0_fe0355c2e6b9.slice/crio-5e817eed7dae0bcc3342468e9ec7686eee83e50a47776e43618092ddb09f4b91 WatchSource:0}: Error finding container 5e817eed7dae0bcc3342468e9ec7686eee83e50a47776e43618092ddb09f4b91: Status 404 returned error can't find the container with id 5e817eed7dae0bcc3342468e9ec7686eee83e50a47776e43618092ddb09f4b91
Jul 09 11:11:58.178408 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.178389    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2dkxw"
Jul 09 11:11:58.185374 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:58.185356    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aee5e83_b717_46bf_b4d8_07a090cde2c3.slice/crio-e03606f02c6818ec0a7a5186495ca5c867ffa44c4ae2c87013908da4e44ee4a3 WatchSource:0}: Error finding container e03606f02c6818ec0a7a5186495ca5c867ffa44c4ae2c87013908da4e44ee4a3: Status 404 returned error can't find the container with id e03606f02c6818ec0a7a5186495ca5c867ffa44c4ae2c87013908da4e44ee4a3
Jul 09 11:11:58.195983 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.195966    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wbjqk"
Jul 09 11:11:58.203626 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:58.203598    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366bda55_c1aa_460a_aa23_ca69ab6852af.slice/crio-f837fb5e172c646947f7c1ad5d11226a73d5e0fd48961da3e1cd7cbe38c44dbc WatchSource:0}: Error finding container f837fb5e172c646947f7c1ad5d11226a73d5e0fd48961da3e1cd7cbe38c44dbc: Status 404 returned error can't find the container with id f837fb5e172c646947f7c1ad5d11226a73d5e0fd48961da3e1cd7cbe38c44dbc
Jul 09 11:11:58.212413 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.212396    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tx572"
Jul 09 11:11:58.216464 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.216449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rcn84"
Jul 09 11:11:58.220084 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:58.220064    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d1d258_b95a_4f9e_88ac_ac35af980a14.slice/crio-6f25b3731e47653c64a9f512226c4d14d0c8cbcea1261199c66c3e142923acc6 WatchSource:0}: Error finding container 6f25b3731e47653c64a9f512226c4d14d0c8cbcea1261199c66c3e142923acc6: Status 404 returned error can't find the container with id 6f25b3731e47653c64a9f512226c4d14d0c8cbcea1261199c66c3e142923acc6
Jul 09 11:11:58.224250 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:11:58.224233    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bc7340_89d1_40c5_9469_8fd2cb0cd905.slice/crio-c1a325709ef1a68473f9bb93a155e1e54b62a607e8bdcea6cf22544e5be4fffb WatchSource:0}: Error finding container c1a325709ef1a68473f9bb93a155e1e54b62a607e8bdcea6cf22544e5be4fffb: Status 404 returned error can't find the container with id c1a325709ef1a68473f9bb93a155e1e54b62a607e8bdcea6cf22544e5be4fffb
Jul 09 11:11:58.225696 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:58.225679    2563 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-multus start failed in pod multus-rcn84_openshift-multus(85bc7340-89d1-40c5-9469-8fd2cb0cd905): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 09 11:11:58.225746 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:58.225707    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-rcn84" podUID="85bc7340-89d1-40c5-9469-8fd2cb0cd905"
Jul 09 11:11:58.325291 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.325255    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 11:11:58.477999 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.477912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs\") pod \"network-metrics-daemon-hphjc\" (UID: \"314935e3-3e91-4cc8-b5fc-15795b54d57a\") " pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:11:58.477999 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.477961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret\") pod \"global-pull-secret-syncer-q7c8l\" (UID: \"346f1026-54ad-4ae6-9134-e2d1a31e565f\") " pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:11:58.478209 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:58.478065    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 11:11:58.478209 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:58.478075    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 11:11:58.478209 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:58.478119    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret podName:346f1026-54ad-4ae6-9134-e2d1a31e565f nodeName:}" failed. No retries permitted until 2026-07-09 11:11:59.478105384 +0000 UTC m=+3.069836543 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret") pod "global-pull-secret-syncer-q7c8l" (UID: "346f1026-54ad-4ae6-9134-e2d1a31e565f") : object "kube-system"/"original-pull-secret" not registered
Jul 09 11:11:58.478209 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:58.478132    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs podName:314935e3-3e91-4cc8-b5fc-15795b54d57a nodeName:}" failed. No retries permitted until 2026-07-09 11:11:59.478127005 +0000 UTC m=+3.069858147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs") pod "network-metrics-daemon-hphjc" (UID: "314935e3-3e91-4cc8-b5fc-15795b54d57a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 11:11:58.578693 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.578646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltgk\" (UniqueName: \"kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk\") pod \"network-check-target-bzxtz\" (UID: \"e3650c56-8363-4f9a-94ec-e5de7bbea057\") " pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:11:58.578876 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:58.578823    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 11:11:58.578876 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:58.578845    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 11:11:58.578876 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:58.578856    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wltgk for pod openshift-network-diagnostics/network-check-target-bzxtz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 11:11:58.579102 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:58.578925    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk podName:e3650c56-8363-4f9a-94ec-e5de7bbea057 nodeName:}" failed. No retries permitted until 2026-07-09 11:11:59.578903792 +0000 UTC m=+3.170634949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wltgk" (UniqueName: "kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk") pod "network-check-target-bzxtz" (UID: "e3650c56-8363-4f9a-94ec-e5de7bbea057") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 11:11:58.719734 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.719700    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 11:11:58.865085 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.865001    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 11:06:57 +0000 UTC" deadline="2027-12-03 13:38:30.155505351 +0000 UTC"
Jul 09 11:11:58.865085 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.865036    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12290h26m31.290474224s"
Jul 09 11:11:58.947082 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.947044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:11:58.947238 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:58.947173    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzxtz" podUID="e3650c56-8363-4f9a-94ec-e5de7bbea057"
Jul 09 11:11:58.963703 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.963645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rzgk" event={"ID":"f1907393-916d-43d8-8ec1-0cb8029ad259","Type":"ContainerStarted","Data":"db6a6db8bd6e6965f247ea169285599e7a5a3a01dbe95647c3a88db89b3a5901"}
Jul 09 11:11:58.967645 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.967617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k7d7z" event={"ID":"72b5c8e9-cd8b-4dde-9850-824fbf0818e7","Type":"ContainerStarted","Data":"ce99ee0f0d9493a0848143a2db108d2fa8d30b2807df23393bd760df1cbd54ca"}
Jul 09 11:11:58.976881 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.976855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97" event={"ID":"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863","Type":"ContainerStarted","Data":"e9dc58499122223857087140337fe3b7ec36d225b7b1b1d975f50d8a7fe9da93"}
Jul 09 11:11:58.994834 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:58.994726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" event={"ID":"2aa1cbda48d30d26959a84b1143a3d09","Type":"ContainerStarted","Data":"10b6f78a1840900fb537fce2baecaa94c4f79de57cc94c735714057747a955ea"}
Jul 09 11:11:59.013356 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:59.013313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" event={"ID":"4aca4cc90e184685e6cdf76e3dd114f1","Type":"ContainerStarted","Data":"3c53da916f3b33516776d610fafaf8355c9e00bc8155951de279a863c2f6b021"}
Jul 09 11:11:59.025615 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:59.025528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tx572" event={"ID":"00d1d258-b95a-4f9e-88ac-ac35af980a14","Type":"ContainerStarted","Data":"6f25b3731e47653c64a9f512226c4d14d0c8cbcea1261199c66c3e142923acc6"}
Jul 09 11:11:59.039763 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:59.039717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wbjqk" event={"ID":"366bda55-c1aa-460a-aa23-ca69ab6852af","Type":"ContainerStarted","Data":"f837fb5e172c646947f7c1ad5d11226a73d5e0fd48961da3e1cd7cbe38c44dbc"}
Jul 09 11:11:59.053054 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:59.052991    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5f2bh" event={"ID":"2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9","Type":"ContainerStarted","Data":"5e817eed7dae0bcc3342468e9ec7686eee83e50a47776e43618092ddb09f4b91"}
Jul 09 11:11:59.065354 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:59.065318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rcn84" event={"ID":"85bc7340-89d1-40c5-9469-8fd2cb0cd905","Type":"ContainerStarted","Data":"c1a325709ef1a68473f9bb93a155e1e54b62a607e8bdcea6cf22544e5be4fffb"}
Jul 09 11:11:59.075015 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:59.074976    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:55ec7147f859301bbb8288966c8a21c5db729200f3c3d91db671978baef79517\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-rcn84" podUID="85bc7340-89d1-40c5-9469-8fd2cb0cd905"
Jul 09 11:11:59.084955 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:59.082871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2dkxw" event={"ID":"0aee5e83-b717-46bf-b4d8-07a090cde2c3","Type":"ContainerStarted","Data":"e03606f02c6818ec0a7a5186495ca5c867ffa44c4ae2c87013908da4e44ee4a3"}
Jul 09 11:11:59.100540 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:59.100420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w" event={"ID":"75dd42d9-d05a-44c9-8d18-664ecaf7feeb","Type":"ContainerStarted","Data":"095d288235bbcc870436bc0fb7b5e85875a318283b76a9621906db7a93952ef7"}
Jul 09 11:11:59.486121 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:59.486030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs\") pod \"network-metrics-daemon-hphjc\" (UID: \"314935e3-3e91-4cc8-b5fc-15795b54d57a\") " pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:11:59.486121 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:59.486100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret\") pod \"global-pull-secret-syncer-q7c8l\" (UID: \"346f1026-54ad-4ae6-9134-e2d1a31e565f\") " pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:11:59.486338 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:59.486246    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 11:11:59.486338 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:59.486309    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret podName:346f1026-54ad-4ae6-9134-e2d1a31e565f nodeName:}" failed. No retries permitted until 2026-07-09 11:12:01.486290993 +0000 UTC m=+5.078022139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret") pod "global-pull-secret-syncer-q7c8l" (UID: "346f1026-54ad-4ae6-9134-e2d1a31e565f") : object "kube-system"/"original-pull-secret" not registered
Jul 09 11:11:59.486706 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:59.486687    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 11:11:59.486802 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:59.486742    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs podName:314935e3-3e91-4cc8-b5fc-15795b54d57a nodeName:}" failed. No retries permitted until 2026-07-09 11:12:01.486725307 +0000 UTC m=+5.078456451 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs") pod "network-metrics-daemon-hphjc" (UID: "314935e3-3e91-4cc8-b5fc-15795b54d57a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 11:11:59.586976 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:59.586935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltgk\" (UniqueName: \"kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk\") pod \"network-check-target-bzxtz\" (UID: \"e3650c56-8363-4f9a-94ec-e5de7bbea057\") " pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:11:59.587156 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:59.587116    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 11:11:59.587156 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:59.587135    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 11:11:59.587156 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:59.587149    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wltgk for pod openshift-network-diagnostics/network-check-target-bzxtz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 11:11:59.587319 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:59.587207    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk podName:e3650c56-8363-4f9a-94ec-e5de7bbea057 nodeName:}" failed. No retries permitted until 2026-07-09 11:12:01.587188067 +0000 UTC m=+5.178919212 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wltgk" (UniqueName: "kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk") pod "network-check-target-bzxtz" (UID: "e3650c56-8363-4f9a-94ec-e5de7bbea057") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 11:11:59.721731 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:59.721503    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 11:11:59.866250 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:59.866141    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 11:06:57 +0000 UTC" deadline="2028-02-20 02:59:34.175278325 +0000 UTC"
Jul 09 11:11:59.866250 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:59.866201    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14175h47m34.30908149s"
Jul 09 11:11:59.945693 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:59.945020    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:11:59.945693 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:59.945152    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:11:59.945693 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:11:59.945531    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:11:59.945693 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:11:59.945630    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:12:00.945236 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:00.945202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:00.945724 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:00.945334    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzxtz" podUID="e3650c56-8363-4f9a-94ec-e5de7bbea057"
Jul 09 11:12:01.503031 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:01.502994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs\") pod \"network-metrics-daemon-hphjc\" (UID: \"314935e3-3e91-4cc8-b5fc-15795b54d57a\") " pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:01.503229 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:01.503062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret\") pod \"global-pull-secret-syncer-q7c8l\" (UID: \"346f1026-54ad-4ae6-9134-e2d1a31e565f\") " pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:01.503229 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:01.503166    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 11:12:01.503229 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:01.503215    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 11:12:01.503383 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:01.503246    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs podName:314935e3-3e91-4cc8-b5fc-15795b54d57a nodeName:}" failed. No retries permitted until 2026-07-09 11:12:05.503224079 +0000 UTC m=+9.094955224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs") pod "network-metrics-daemon-hphjc" (UID: "314935e3-3e91-4cc8-b5fc-15795b54d57a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 11:12:01.503383 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:01.503272    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret podName:346f1026-54ad-4ae6-9134-e2d1a31e565f nodeName:}" failed. No retries permitted until 2026-07-09 11:12:05.503254441 +0000 UTC m=+9.094985589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret") pod "global-pull-secret-syncer-q7c8l" (UID: "346f1026-54ad-4ae6-9134-e2d1a31e565f") : object "kube-system"/"original-pull-secret" not registered
Jul 09 11:12:01.604041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:01.603986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltgk\" (UniqueName: \"kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk\") pod \"network-check-target-bzxtz\" (UID: \"e3650c56-8363-4f9a-94ec-e5de7bbea057\") " pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:01.604296 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:01.604260    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 11:12:01.604296 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:01.604290    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 11:12:01.604296 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:01.604303    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wltgk for pod openshift-network-diagnostics/network-check-target-bzxtz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 11:12:01.604559 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:01.604374    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk podName:e3650c56-8363-4f9a-94ec-e5de7bbea057 nodeName:}" failed. No retries permitted until 2026-07-09 11:12:05.60435617 +0000 UTC m=+9.196087333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wltgk" (UniqueName: "kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk") pod "network-check-target-bzxtz" (UID: "e3650c56-8363-4f9a-94ec-e5de7bbea057") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 11:12:01.945942 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:01.945383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:01.945942 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:01.945388    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:01.945942 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:01.945536    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:12:01.945942 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:01.945600    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:12:02.945263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:02.945210    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:02.945406 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:02.945388    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzxtz" podUID="e3650c56-8363-4f9a-94ec-e5de7bbea057"
Jul 09 11:12:03.945059 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:03.945026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:03.945059 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:03.945058    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:03.945573 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:03.945159    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:12:03.945573 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:03.945282    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:12:04.945381 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:04.945350    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:04.945872 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:04.945479    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzxtz" podUID="e3650c56-8363-4f9a-94ec-e5de7bbea057"
Jul 09 11:12:05.532518 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:05.532484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs\") pod \"network-metrics-daemon-hphjc\" (UID: \"314935e3-3e91-4cc8-b5fc-15795b54d57a\") " pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:05.532719 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:05.532540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret\") pod \"global-pull-secret-syncer-q7c8l\" (UID: \"346f1026-54ad-4ae6-9134-e2d1a31e565f\") " pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:05.532719 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:05.532649    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 11:12:05.532719 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:05.532654    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 11:12:05.532719 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:05.532706    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret podName:346f1026-54ad-4ae6-9134-e2d1a31e565f nodeName:}" failed. No retries permitted until 2026-07-09 11:12:13.532690409 +0000 UTC m=+17.124421551 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret") pod "global-pull-secret-syncer-q7c8l" (UID: "346f1026-54ad-4ae6-9134-e2d1a31e565f") : object "kube-system"/"original-pull-secret" not registered
Jul 09 11:12:05.532719 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:05.532721    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs podName:314935e3-3e91-4cc8-b5fc-15795b54d57a nodeName:}" failed. No retries permitted until 2026-07-09 11:12:13.532714674 +0000 UTC m=+17.124445815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs") pod "network-metrics-daemon-hphjc" (UID: "314935e3-3e91-4cc8-b5fc-15795b54d57a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 11:12:05.632970 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:05.632916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltgk\" (UniqueName: \"kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk\") pod \"network-check-target-bzxtz\" (UID: \"e3650c56-8363-4f9a-94ec-e5de7bbea057\") " pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:05.633167 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:05.633106    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 11:12:05.633167 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:05.633134    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 11:12:05.633167 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:05.633148    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wltgk for pod openshift-network-diagnostics/network-check-target-bzxtz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 11:12:05.633340 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:05.633215    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk podName:e3650c56-8363-4f9a-94ec-e5de7bbea057 nodeName:}" failed. No retries permitted until 2026-07-09 11:12:13.63319498 +0000 UTC m=+17.224926144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wltgk" (UniqueName: "kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk") pod "network-check-target-bzxtz" (UID: "e3650c56-8363-4f9a-94ec-e5de7bbea057") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 11:12:05.945518 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:05.945444    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:05.945951 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:05.945573    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:12:05.945951 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:05.945626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:05.945951 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:05.945720    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:12:06.945869 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:06.945834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:06.946331 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:06.945941    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzxtz" podUID="e3650c56-8363-4f9a-94ec-e5de7bbea057"
Jul 09 11:12:07.944457 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:07.944423    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:07.944644 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:07.944438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:07.944644 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:07.944556    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:12:07.944766 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:07.944671    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:12:08.945431 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:08.945396    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:08.945902 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:08.945526    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzxtz" podUID="e3650c56-8363-4f9a-94ec-e5de7bbea057"
Jul 09 11:12:09.945172 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:09.945127    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:09.945172 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:09.945160    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:09.945412 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:09.945271    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:12:09.945473 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:09.945418    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:12:10.945118 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:10.945084    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:10.945305 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:10.945210    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzxtz" podUID="e3650c56-8363-4f9a-94ec-e5de7bbea057"
Jul 09 11:12:11.945259 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:11.945220    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:11.945259 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:11.945220    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:11.945796 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:11.945496    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:12:11.945796 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:11.945594    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:12:12.944575 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:12.944550    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:12.944695 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:12.944654    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzxtz" podUID="e3650c56-8363-4f9a-94ec-e5de7bbea057"
Jul 09 11:12:13.132249 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:13.132211    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" event={"ID":"4aca4cc90e184685e6cdf76e3dd114f1","Type":"ContainerStarted","Data":"4c2f4a0899f86939b302f865a73c6feb89023fbb6e3027f152d4832c0c672c8c"}
Jul 09 11:12:13.134043 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:13.134006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wbjqk" event={"ID":"366bda55-c1aa-460a-aa23-ca69ab6852af","Type":"ContainerStarted","Data":"f12fcc911274ee35a1abef5cf645b3ce5a811d16aa3153f7346762e0f47cd13d"}
Jul 09 11:12:13.135717 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:13.135683    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w" event={"ID":"75dd42d9-d05a-44c9-8d18-664ecaf7feeb","Type":"ContainerStarted","Data":"a815c398ec4489f28a63592a1ae288d591f3251cd46c0a0d6ba38fef601dfb90"}
Jul 09 11:12:13.135829 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:13.135761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w" event={"ID":"75dd42d9-d05a-44c9-8d18-664ecaf7feeb","Type":"ContainerStarted","Data":"899e298b6681b80bd667360f67f9e35ec8ec43403ed8d8ecfc7332bf89714d4e"}
Jul 09 11:12:13.148124 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:13.148071    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-176.ec2.internal" podStartSLOduration=16.14805324 podStartE2EDuration="16.14805324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 11:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:12:13.147561833 +0000 UTC m=+16.739292997" watchObservedRunningTime="2026-07-09 11:12:13.14805324 +0000 UTC m=+16.739784407"
Jul 09 11:12:13.168098 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:13.168040    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wbjqk" podStartSLOduration=1.779966068 podStartE2EDuration="16.168020442s" totalImagesPullingTime="14.388054374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 11:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:12:13.167074626 +0000 UTC m=+16.758805792" watchObservedRunningTime="2026-07-09 11:12:13.168020442 +0000 UTC m=+16.759751607"
Jul 09 11:12:13.590765 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:13.590586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs\") pod \"network-metrics-daemon-hphjc\" (UID: \"314935e3-3e91-4cc8-b5fc-15795b54d57a\") " pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:13.590895 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:13.590690    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 11:12:13.590895 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:13.590814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret\") pod \"global-pull-secret-syncer-q7c8l\" (UID: \"346f1026-54ad-4ae6-9134-e2d1a31e565f\") " pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:13.590895 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:13.590862    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs podName:314935e3-3e91-4cc8-b5fc-15795b54d57a nodeName:}" failed. No retries permitted until 2026-07-09 11:12:29.59083822 +0000 UTC m=+33.182569382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs") pod "network-metrics-daemon-hphjc" (UID: "314935e3-3e91-4cc8-b5fc-15795b54d57a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 11:12:13.591052 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:13.590939    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 11:12:13.591052 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:13.590995    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret podName:346f1026-54ad-4ae6-9134-e2d1a31e565f nodeName:}" failed. No retries permitted until 2026-07-09 11:12:29.590979572 +0000 UTC m=+33.182710720 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret") pod "global-pull-secret-syncer-q7c8l" (UID: "346f1026-54ad-4ae6-9134-e2d1a31e565f") : object "kube-system"/"original-pull-secret" not registered
Jul 09 11:12:13.692019 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:13.691939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltgk\" (UniqueName: \"kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk\") pod \"network-check-target-bzxtz\" (UID: \"e3650c56-8363-4f9a-94ec-e5de7bbea057\") " pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:13.692196 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:13.692092    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 11:12:13.692196 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:13.692137    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 11:12:13.692196 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:13.692152    2563 projected.go:196] Error preparing data for projected volume kube-api-access-wltgk for pod openshift-network-diagnostics/network-check-target-bzxtz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 11:12:13.692372 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:13.692220    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk podName:e3650c56-8363-4f9a-94ec-e5de7bbea057 nodeName:}" failed. No retries permitted until 2026-07-09 11:12:29.692200873 +0000 UTC m=+33.283932023 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wltgk" (UniqueName: "kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk") pod "network-check-target-bzxtz" (UID: "e3650c56-8363-4f9a-94ec-e5de7bbea057") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 11:12:13.944637 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:13.944587    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:13.944863 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:13.944647    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:13.944863 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:13.944749    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:12:13.944863 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:13.944846    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:12:14.139593 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.139501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2dkxw" event={"ID":"0aee5e83-b717-46bf-b4d8-07a090cde2c3","Type":"ContainerStarted","Data":"eece9822293b91cdcc7719483b9f3e886ff0de88bb2c8429367ea22c29f8cc82"}
Jul 09 11:12:14.142718 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.142685    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:12:14.143089 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.143068    2563 generic.go:356] "Generic (PLEG): container finished" podID="75dd42d9-d05a-44c9-8d18-664ecaf7feeb" containerID="a815c398ec4489f28a63592a1ae288d591f3251cd46c0a0d6ba38fef601dfb90" exitCode=1
Jul 09 11:12:14.143180 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.143112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w" event={"ID":"75dd42d9-d05a-44c9-8d18-664ecaf7feeb","Type":"ContainerDied","Data":"a815c398ec4489f28a63592a1ae288d591f3251cd46c0a0d6ba38fef601dfb90"}
Jul 09 11:12:14.144107 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.143546    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w" event={"ID":"75dd42d9-d05a-44c9-8d18-664ecaf7feeb","Type":"ContainerStarted","Data":"903fdd63369e7693de1797ef7f3250329dcfde853dff2ae19d30d08998dc1d51"}
Jul 09 11:12:14.144107 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.143633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w" event={"ID":"75dd42d9-d05a-44c9-8d18-664ecaf7feeb","Type":"ContainerStarted","Data":"7f1fe4f41d0dc6e9ad7e7c0a5597f6e7de797fe220808e1c2c9b48d529d4ef9c"}
Jul 09 11:12:14.145803 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.144375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w" event={"ID":"75dd42d9-d05a-44c9-8d18-664ecaf7feeb","Type":"ContainerStarted","Data":"1132b0cdbb19770453fd5bf6a3ef46ae17fa4417e62067ab20dbf2c4c7ab1bfc"}
Jul 09 11:12:14.145803 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.144464    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w" event={"ID":"75dd42d9-d05a-44c9-8d18-664ecaf7feeb","Type":"ContainerStarted","Data":"f6d96263b17a03f65fd439290b936bddae38b3a1c98c85708baa266334483652"}
Jul 09 11:12:14.147664 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.147640    2563 generic.go:356] "Generic (PLEG): container finished" podID="f1907393-916d-43d8-8ec1-0cb8029ad259" containerID="a5184d64641fb3bbac2465c1395c13138d9f48ffbe1a4c9f97c834d16e288813" exitCode=0
Jul 09 11:12:14.147757 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.147723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rzgk" event={"ID":"f1907393-916d-43d8-8ec1-0cb8029ad259","Type":"ContainerDied","Data":"a5184d64641fb3bbac2465c1395c13138d9f48ffbe1a4c9f97c834d16e288813"}
Jul 09 11:12:14.149461 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.149395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k7d7z" event={"ID":"72b5c8e9-cd8b-4dde-9850-824fbf0818e7","Type":"ContainerStarted","Data":"96a599dc569cd674ba339e808d62354aaf1acd7bcba7e1d030ed89e85728da9f"}
Jul 09 11:12:14.150907 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.150889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97" event={"ID":"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863","Type":"ContainerStarted","Data":"1ec3d1bf8161bc6ef63fc96f4851cb3f1f1c6f153eb66a5cfc1c2c09081e8ac7"}
Jul 09 11:12:14.152393 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.152371    2563 generic.go:356] "Generic (PLEG): container finished" podID="2aa1cbda48d30d26959a84b1143a3d09" containerID="5d844113debc8a86af3533ee6e5aa43ab55aadb34391703f717ed963b75e33fc" exitCode=0
Jul 09 11:12:14.152501 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.152453    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" event={"ID":"2aa1cbda48d30d26959a84b1143a3d09","Type":"ContainerDied","Data":"5d844113debc8a86af3533ee6e5aa43ab55aadb34391703f717ed963b75e33fc"}
Jul 09 11:12:14.154014 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.153992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tx572" event={"ID":"00d1d258-b95a-4f9e-88ac-ac35af980a14","Type":"ContainerStarted","Data":"39c061307e12af02f098b1925844401623e8959e4ae87e6a9067c34d21bcd9f5"}
Jul 09 11:12:14.155822 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.155796    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5f2bh" event={"ID":"2a5adaeb-23e5-47a8-a3a0-fe0355c2e6b9","Type":"ContainerStarted","Data":"58b1b94b3042a43151c9287939d8c99ed49f2916c83780a3a9d9024814d7d292"}
Jul 09 11:12:14.162681 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.162632    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2dkxw" podStartSLOduration=2.787249344 podStartE2EDuration="17.16261509s" totalImagesPullingTime="14.375365746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:12:14.162023695 +0000 UTC m=+17.753754860" watchObservedRunningTime="2026-07-09 11:12:14.16261509 +0000 UTC m=+17.754346256"
Jul 09 11:12:14.190515 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.190445    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tx572" podStartSLOduration=2.832868516 podStartE2EDuration="17.190427489s" totalImagesPullingTime="14.357558973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 11:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:12:14.190274962 +0000 UTC m=+17.782006165" watchObservedRunningTime="2026-07-09 11:12:14.190427489 +0000 UTC m=+17.782158654"
Jul 09 11:12:14.243723 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.243461    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:14.278352 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.278298    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-k7d7z" podStartSLOduration=2.845512184 podStartE2EDuration="17.278276165s" totalImagesPullingTime="14.432763981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 11:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:12:14.244091211 +0000 UTC m=+17.835822377" watchObservedRunningTime="2026-07-09 11:12:14.278276165 +0000 UTC m=+17.870007330"
Jul 09 11:12:14.299575 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.299525    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5f2bh" podStartSLOduration=2.878703396 podStartE2EDuration="17.299502849s" totalImagesPullingTime="14.420799453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 11:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:12:14.298841803 +0000 UTC m=+17.890572968" watchObservedRunningTime="2026-07-09 11:12:14.299502849 +0000 UTC m=+17.891234014"
Jul 09 11:12:14.419639 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.419532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:14.516407 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.516346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:14.652292 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.652231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:14.683755 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.683732    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 11:12:14.831010 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.830858    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T11:12:14.68375246Z","UUID":"9700929e-d2ca-491e-90b0-6bfca36be680","Handler":null,"Name":"","Endpoint":""}
Jul 09 11:12:14.833474 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.833450    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 11:12:14.833626 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.833498    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 11:12:14.945052 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:14.944964    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:14.945209 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:14.945095    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzxtz" podUID="e3650c56-8363-4f9a-94ec-e5de7bbea057"
Jul 09 11:12:15.065823 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:15.065773    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:15.133026 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:15.132989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:15.162426 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:15.162390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97" event={"ID":"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863","Type":"ContainerStarted","Data":"6f528fb64bd50755179947d1aeecee986349e4b6f6546afa52868626e51d09f3"}
Jul 09 11:12:15.165192 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:15.165161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" event={"ID":"2aa1cbda48d30d26959a84b1143a3d09","Type":"ContainerStarted","Data":"55d4f31f3a22c799b1e8a45c144623ddce6d813f0a97dc591cee7bb0a4b75180"}
Jul 09 11:12:15.192052 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:15.191992    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-176.ec2.internal" podStartSLOduration=18.191972275 podStartE2EDuration="18.191972275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 11:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:12:15.190973757 +0000 UTC m=+18.782704922" watchObservedRunningTime="2026-07-09 11:12:15.191972275 +0000 UTC m=+18.783703441"
Jul 09 11:12:15.945052 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:15.944972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:15.945052 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:15.945012    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:15.945232 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:15.945110    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:12:15.945280 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:15.945230    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:12:16.171890 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:16.171809    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:12:16.172591 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:16.172548    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w" event={"ID":"75dd42d9-d05a-44c9-8d18-664ecaf7feeb","Type":"ContainerStarted","Data":"8547e467d230dbe905fb15054abed603c3f764867d99728b84a8c1e8b625548a"}
Jul 09 11:12:16.175340 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:16.175310    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97" event={"ID":"2a0b9e74-a148-46e7-9cd9-4ffd6acd3863","Type":"ContainerStarted","Data":"6493ab9265a9558b5e3bb201d3fff150a2d4fcea7e8060d9a1c4f5d567c10f61"}
Jul 09 11:12:16.946194 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:16.946162    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:16.946372 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:16.946264    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzxtz" podUID="e3650c56-8363-4f9a-94ec-e5de7bbea057"
Jul 09 11:12:17.944655 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:17.944623    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:17.945136 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:17.944661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:17.945136 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:17.944747    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:12:17.945136 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:17.944886    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:12:18.550705 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:18.550673    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2dkxw"
Jul 09 11:12:18.551427 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:18.551395    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2dkxw"
Jul 09 11:12:18.566621 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:18.566574    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xvq97" podStartSLOduration=10.635764831 podStartE2EDuration="22.56655947s" totalImagesPullingTime="11.930794639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 11:11:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:12:16.198357581 +0000 UTC m=+19.790088749" watchObservedRunningTime="2026-07-09 11:12:18.56655947 +0000 UTC m=+22.158290662"
Jul 09 11:12:18.945274 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:18.945048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:18.945665 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:18.945372    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzxtz" podUID="e3650c56-8363-4f9a-94ec-e5de7bbea057"
Jul 09 11:12:19.186955 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:19.186885    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:12:19.187443 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:19.187399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w" event={"ID":"75dd42d9-d05a-44c9-8d18-664ecaf7feeb","Type":"ContainerStarted","Data":"32944c9d370af032be3d359d8edc22a1bf59682b105e29b1322004c82fbb8450"}
Jul 09 11:12:19.187690 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:19.187670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:12:19.187945 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:19.187927    2563 scope.go:117] "RemoveContainer" containerID="a815c398ec4489f28a63592a1ae288d591f3251cd46c0a0d6ba38fef601dfb90"
Jul 09 11:12:19.189601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:19.189578    2563 generic.go:356] "Generic (PLEG): container finished" podID="f1907393-916d-43d8-8ec1-0cb8029ad259" containerID="8d36d31368efbd3c64054e241a3b5fb859d7e602f7752cd1dbc9a50691de0aa9" exitCode=0
Jul 09 11:12:19.189816 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:19.189736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rzgk" event={"ID":"f1907393-916d-43d8-8ec1-0cb8029ad259","Type":"ContainerDied","Data":"8d36d31368efbd3c64054e241a3b5fb859d7e602f7752cd1dbc9a50691de0aa9"}
Jul 09 11:12:19.190052 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:19.190014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2dkxw"
Jul 09 11:12:19.190458 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:19.190437    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2dkxw"
Jul 09 11:12:19.206589 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:19.206562    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:12:19.343482 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:19.343441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:19.945450 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:19.944987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:19.945450 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:19.945144    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:12:19.945450 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:19.945211    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:19.945450 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:19.945436    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:12:19.968357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:19.968322    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:20.072432 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:20.072127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:20.196565 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:20.196484    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:12:20.196938 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:20.196904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w" event={"ID":"75dd42d9-d05a-44c9-8d18-664ecaf7feeb","Type":"ContainerStarted","Data":"ce231264cea6fc2fb2721879cc8aaa5104d56243fbcd9189cf035998f3307877"}
Jul 09 11:12:20.197335 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:20.197317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:12:20.197465 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:20.197392    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:12:20.199870 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:20.199666    2563 generic.go:356] "Generic (PLEG): container finished" podID="f1907393-916d-43d8-8ec1-0cb8029ad259" containerID="9bc67033a3e79650d7ac695ad892e32e352afa53144d347af37011947de82dbe" exitCode=0
Jul 09 11:12:20.199870 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:20.199771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rzgk" event={"ID":"f1907393-916d-43d8-8ec1-0cb8029ad259","Type":"ContainerDied","Data":"9bc67033a3e79650d7ac695ad892e32e352afa53144d347af37011947de82dbe"}
Jul 09 11:12:20.216080 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:20.216045    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:12:20.282648 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:20.282600    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w" podStartSLOduration=8.550426182 podStartE2EDuration="23.282582099s" totalImagesPullingTime="14.732155917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 11:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:12:20.23789728 +0000 UTC m=+23.829628447" watchObservedRunningTime="2026-07-09 11:12:20.282582099 +0000 UTC m=+23.874313265"
Jul 09 11:12:20.735425 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:20.735394    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bzxtz"]
Jul 09 11:12:20.735581 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:20.735549    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:20.735678 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:20.735652    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzxtz" podUID="e3650c56-8363-4f9a-94ec-e5de7bbea057"
Jul 09 11:12:20.736256 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:20.736231    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hphjc"]
Jul 09 11:12:20.736377 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:20.736348    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:20.736480 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:20.736455    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:12:20.737726 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:20.737692    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q7c8l"]
Jul 09 11:12:20.737837 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:20.737808    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:20.737918 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:20.737897    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:12:21.944925 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:21.944896    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:21.945451 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:21.945036    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:12:22.945604 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:22.945393    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:22.946035 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:22.945454    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:22.946035 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:22.945726    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:12:22.946035 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:22.945807    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzxtz" podUID="e3650c56-8363-4f9a-94ec-e5de7bbea057"
Jul 09 11:12:23.944888 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:23.944816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:23.945029 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:23.944929    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:12:24.944746 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:24.944718    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:24.945176 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:24.944836    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:12:24.945176 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:24.944905    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:24.945176 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:24.944953    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bzxtz" podUID="e3650c56-8363-4f9a-94ec-e5de7bbea057"
Jul 09 11:12:25.213202 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:25.213177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rcn84" event={"ID":"85bc7340-89d1-40c5-9469-8fd2cb0cd905","Type":"ContainerStarted","Data":"c3084db7bf2319025f42cc21bbd26d760b00184931ecd607e5411a94f85d075a"}
Jul 09 11:12:25.230869 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:25.230769    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rcn84" podStartSLOduration=1.968157219 podStartE2EDuration="28.230754558s" totalImagesPullingTime="26.262597339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 11:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-09 11:12:25.230424965 +0000 UTC m=+28.822156131" watchObservedRunningTime="2026-07-09 11:12:25.230754558 +0000 UTC m=+28.822485722"
Jul 09 11:12:25.353120 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:25.353086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:25.421086 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:25.421001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:25.945245 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:25.945216    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:25.945597 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:25.945332    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7c8l" podUID="346f1026-54ad-4ae6-9134-e2d1a31e565f"
Jul 09 11:12:26.217940 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.217852    2563 generic.go:356] "Generic (PLEG): container finished" podID="f1907393-916d-43d8-8ec1-0cb8029ad259" containerID="5a5a8e0c3cfb57ca8088c639cf76c93561674466df5157614199f57728bf306c" exitCode=0
Jul 09 11:12:26.218081 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.217941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rzgk" event={"ID":"f1907393-916d-43d8-8ec1-0cb8029ad259","Type":"ContainerDied","Data":"5a5a8e0c3cfb57ca8088c639cf76c93561674466df5157614199f57728bf306c"}
Jul 09 11:12:26.260800 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.260758    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-176.ec2.internal" event="NodeReady"
Jul 09 11:12:26.260932 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.260900    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 11:12:26.303237 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.303204    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fmjjv"]
Jul 09 11:12:26.337350 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.337323    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-745gr"]
Jul 09 11:12:26.337489 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.337435    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:26.340191 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.340171    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 11:12:26.340191 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.340181    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6pmqk\""
Jul 09 11:12:26.340372 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.340239    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 11:12:26.359066 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.359040    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fmjjv"]
Jul 09 11:12:26.359164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.359074    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-745gr"]
Jul 09 11:12:26.359164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.359147    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-745gr"
Jul 09 11:12:26.361827 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.361804    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 11:12:26.361930 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.361807    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 11:12:26.361930 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.361841    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 11:12:26.361930 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.361867    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8jsqk\""
Jul 09 11:12:26.390647 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.390609    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49b90316-83e9-4bc1-a9b3-e118c434d9fd-config-volume\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:26.390647 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.390648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:26.390868 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.390759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b90316-83e9-4bc1-a9b3-e118c434d9fd-tmp-dir\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:26.390868 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.390801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z4w2\" (UniqueName: \"kubernetes.io/projected/49b90316-83e9-4bc1-a9b3-e118c434d9fd-kube-api-access-7z4w2\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:26.448064 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.448027    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:26.492164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.492076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49b90316-83e9-4bc1-a9b3-e118c434d9fd-config-volume\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:26.492164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.492122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:26.492164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.492143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53a84cad-5a9e-4284-95f4-7ed61d11ac64-cert\") pod \"ingress-canary-745gr\" (UID: \"53a84cad-5a9e-4284-95f4-7ed61d11ac64\") " pod="openshift-ingress-canary/ingress-canary-745gr"
Jul 09 11:12:26.492438 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.492173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncz2m\" (UniqueName: \"kubernetes.io/projected/53a84cad-5a9e-4284-95f4-7ed61d11ac64-kube-api-access-ncz2m\") pod \"ingress-canary-745gr\" (UID: \"53a84cad-5a9e-4284-95f4-7ed61d11ac64\") " pod="openshift-ingress-canary/ingress-canary-745gr"
Jul 09 11:12:26.492438 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.492223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b90316-83e9-4bc1-a9b3-e118c434d9fd-tmp-dir\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:26.492438 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.492247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4w2\" (UniqueName: \"kubernetes.io/projected/49b90316-83e9-4bc1-a9b3-e118c434d9fd-kube-api-access-7z4w2\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:26.492438 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:26.492257    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 11:12:26.492438 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:26.492422    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls podName:49b90316-83e9-4bc1-a9b3-e118c434d9fd nodeName:}" failed. No retries permitted until 2026-07-09 11:12:26.99239972 +0000 UTC m=+30.584130872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls") pod "dns-default-fmjjv" (UID: "49b90316-83e9-4bc1-a9b3-e118c434d9fd") : secret "dns-default-metrics-tls" not found
Jul 09 11:12:26.492694 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.492533    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49b90316-83e9-4bc1-a9b3-e118c434d9fd-tmp-dir\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:26.492694 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.492662    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49b90316-83e9-4bc1-a9b3-e118c434d9fd-config-volume\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:26.503272 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.503159    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z4w2\" (UniqueName: \"kubernetes.io/projected/49b90316-83e9-4bc1-a9b3-e118c434d9fd-kube-api-access-7z4w2\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:26.550601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.550563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:26.593072 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.593040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncz2m\" (UniqueName: \"kubernetes.io/projected/53a84cad-5a9e-4284-95f4-7ed61d11ac64-kube-api-access-ncz2m\") pod \"ingress-canary-745gr\" (UID: \"53a84cad-5a9e-4284-95f4-7ed61d11ac64\") " pod="openshift-ingress-canary/ingress-canary-745gr"
Jul 09 11:12:26.593234 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.593147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53a84cad-5a9e-4284-95f4-7ed61d11ac64-cert\") pod \"ingress-canary-745gr\" (UID: \"53a84cad-5a9e-4284-95f4-7ed61d11ac64\") " pod="openshift-ingress-canary/ingress-canary-745gr"
Jul 09 11:12:26.595597 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.595575    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53a84cad-5a9e-4284-95f4-7ed61d11ac64-cert\") pod \"ingress-canary-745gr\" (UID: \"53a84cad-5a9e-4284-95f4-7ed61d11ac64\") " pod="openshift-ingress-canary/ingress-canary-745gr"
Jul 09 11:12:26.601350 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.601330    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncz2m\" (UniqueName: \"kubernetes.io/projected/53a84cad-5a9e-4284-95f4-7ed61d11ac64-kube-api-access-ncz2m\") pod \"ingress-canary-745gr\" (UID: \"53a84cad-5a9e-4284-95f4-7ed61d11ac64\") " pod="openshift-ingress-canary/ingress-canary-745gr"
Jul 09 11:12:26.667492 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.667460    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-745gr"
Jul 09 11:12:26.859709 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.859680    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-745gr"]
Jul 09 11:12:26.863484 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:12:26.863455    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a84cad_5a9e_4284_95f4_7ed61d11ac64.slice/crio-742213f811b60da1a19ba358bcf5a3550e107d6bddb9433566df8f5e829d0376 WatchSource:0}: Error finding container 742213f811b60da1a19ba358bcf5a3550e107d6bddb9433566df8f5e829d0376: Status 404 returned error can't find the container with id 742213f811b60da1a19ba358bcf5a3550e107d6bddb9433566df8f5e829d0376
Jul 09 11:12:26.946021 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.945986    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:26.946987 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.946148    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:26.949257 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.949107    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 11:12:26.949257 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.949137    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dn2v5\""
Jul 09 11:12:26.949257 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.949185    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 11:12:26.949257 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.949189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h6b4g\""
Jul 09 11:12:26.949257 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.949137    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 11:12:26.996133 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:26.996096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:26.996321 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:26.996280    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 11:12:26.996373 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:26.996365    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls podName:49b90316-83e9-4bc1-a9b3-e118c434d9fd nodeName:}" failed. No retries permitted until 2026-07-09 11:12:27.9963435 +0000 UTC m=+31.588074659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls") pod "dns-default-fmjjv" (UID: "49b90316-83e9-4bc1-a9b3-e118c434d9fd") : secret "dns-default-metrics-tls" not found
Jul 09 11:12:27.221873 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:27.221796    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-745gr" event={"ID":"53a84cad-5a9e-4284-95f4-7ed61d11ac64","Type":"ContainerStarted","Data":"742213f811b60da1a19ba358bcf5a3550e107d6bddb9433566df8f5e829d0376"}
Jul 09 11:12:27.944800 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:27.944762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:27.947644 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:27.947606    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 11:12:28.005400 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:28.005141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:28.005400 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:28.005292    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 11:12:28.005400 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:28.005374    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls podName:49b90316-83e9-4bc1-a9b3-e118c434d9fd nodeName:}" failed. No retries permitted until 2026-07-09 11:12:30.005352677 +0000 UTC m=+33.597083840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls") pod "dns-default-fmjjv" (UID: "49b90316-83e9-4bc1-a9b3-e118c434d9fd") : secret "dns-default-metrics-tls" not found
Jul 09 11:12:29.618472 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:29.618203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs\") pod \"network-metrics-daemon-hphjc\" (UID: \"314935e3-3e91-4cc8-b5fc-15795b54d57a\") " pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:12:29.618905 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:29.618499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret\") pod \"global-pull-secret-syncer-q7c8l\" (UID: \"346f1026-54ad-4ae6-9134-e2d1a31e565f\") " pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:29.618905 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:29.618354    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 11:12:29.618905 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:29.618616    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs podName:314935e3-3e91-4cc8-b5fc-15795b54d57a nodeName:}" failed. No retries permitted until 2026-07-09 11:13:01.618593518 +0000 UTC m=+65.210324673 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs") pod "network-metrics-daemon-hphjc" (UID: "314935e3-3e91-4cc8-b5fc-15795b54d57a") : secret "metrics-daemon-secret" not found
Jul 09 11:12:29.622309 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:29.622284    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/346f1026-54ad-4ae6-9134-e2d1a31e565f-original-pull-secret\") pod \"global-pull-secret-syncer-q7c8l\" (UID: \"346f1026-54ad-4ae6-9134-e2d1a31e565f\") " pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:29.719773 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:29.719736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltgk\" (UniqueName: \"kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk\") pod \"network-check-target-bzxtz\" (UID: \"e3650c56-8363-4f9a-94ec-e5de7bbea057\") " pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:29.722652 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:29.722625    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wltgk\" (UniqueName: \"kubernetes.io/projected/e3650c56-8363-4f9a-94ec-e5de7bbea057-kube-api-access-wltgk\") pod \"network-check-target-bzxtz\" (UID: \"e3650c56-8363-4f9a-94ec-e5de7bbea057\") " pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:29.757585 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:29.757533    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7c8l"
Jul 09 11:12:29.964673 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:29.964554    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:30.022286 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:30.022251    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:30.022460 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:30.022395    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 11:12:30.022531 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:30.022471    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls podName:49b90316-83e9-4bc1-a9b3-e118c434d9fd nodeName:}" failed. No retries permitted until 2026-07-09 11:12:34.022450587 +0000 UTC m=+37.614181742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls") pod "dns-default-fmjjv" (UID: "49b90316-83e9-4bc1-a9b3-e118c434d9fd") : secret "dns-default-metrics-tls" not found
Jul 09 11:12:31.396655 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:31.396616    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:31.500821 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:31.500606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:31.605034 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:31.605003    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bzxtz"]
Jul 09 11:12:31.608993 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:12:31.608964    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3650c56_8363_4f9a_94ec_e5de7bbea057.slice/crio-7a447e736f83f04c7508e3b1d854a40ff04d036d3d1c2c13aed32adbc9dc8d5f WatchSource:0}: Error finding container 7a447e736f83f04c7508e3b1d854a40ff04d036d3d1c2c13aed32adbc9dc8d5f: Status 404 returned error can't find the container with id 7a447e736f83f04c7508e3b1d854a40ff04d036d3d1c2c13aed32adbc9dc8d5f
Jul 09 11:12:31.620645 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:31.620621    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q7c8l"]
Jul 09 11:12:31.623682 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:12:31.623660    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod346f1026_54ad_4ae6_9134_e2d1a31e565f.slice/crio-8aa1d9d2973558f847cede58fe1733f235f33f6dae1748cc4440d2bc2cdbda6d WatchSource:0}: Error finding container 8aa1d9d2973558f847cede58fe1733f235f33f6dae1748cc4440d2bc2cdbda6d: Status 404 returned error can't find the container with id 8aa1d9d2973558f847cede58fe1733f235f33f6dae1748cc4440d2bc2cdbda6d
Jul 09 11:12:32.235328 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:32.235286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q7c8l" event={"ID":"346f1026-54ad-4ae6-9134-e2d1a31e565f","Type":"ContainerStarted","Data":"8aa1d9d2973558f847cede58fe1733f235f33f6dae1748cc4440d2bc2cdbda6d"}
Jul 09 11:12:32.238229 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:32.238200    2563 generic.go:356] "Generic (PLEG): container finished" podID="f1907393-916d-43d8-8ec1-0cb8029ad259" containerID="85d3b23eb930cacedbcfa1cf010679f55dda5e52175b5782c72080cd0b09c11c" exitCode=0
Jul 09 11:12:32.238362 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:32.238306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rzgk" event={"ID":"f1907393-916d-43d8-8ec1-0cb8029ad259","Type":"ContainerDied","Data":"85d3b23eb930cacedbcfa1cf010679f55dda5e52175b5782c72080cd0b09c11c"}
Jul 09 11:12:32.240514 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:32.240370    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-745gr" event={"ID":"53a84cad-5a9e-4284-95f4-7ed61d11ac64","Type":"ContainerStarted","Data":"80ccfd33efdaa5e39a5062bc6158a5936a6e03f64bc56b8010268281d5d41266"}
Jul 09 11:12:32.241729 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:32.241704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bzxtz" event={"ID":"e3650c56-8363-4f9a-94ec-e5de7bbea057","Type":"ContainerStarted","Data":"7a447e736f83f04c7508e3b1d854a40ff04d036d3d1c2c13aed32adbc9dc8d5f"}
Jul 09 11:12:32.286144 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:32.286090    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-745gr" podStartSLOduration=1.694743272 podStartE2EDuration="6.286067592s" totalImagesPullingTime="4.59132432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:12:32.28527968 +0000 UTC m=+35.877010845" watchObservedRunningTime="2026-07-09 11:12:32.286067592 +0000 UTC m=+35.877798738"
Jul 09 11:12:33.249031 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:33.248994    2563 generic.go:356] "Generic (PLEG): container finished" podID="f1907393-916d-43d8-8ec1-0cb8029ad259" containerID="cd8e641972fcf19292971f37535b4781c1472327b011050aa70d52966fb5e3b5" exitCode=0
Jul 09 11:12:33.249569 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:33.249521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rzgk" event={"ID":"f1907393-916d-43d8-8ec1-0cb8029ad259","Type":"ContainerDied","Data":"cd8e641972fcf19292971f37535b4781c1472327b011050aa70d52966fb5e3b5"}
Jul 09 11:12:34.054987 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:34.054945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:34.055176 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:34.055112    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 11:12:34.055235 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:34.055194    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls podName:49b90316-83e9-4bc1-a9b3-e118c434d9fd nodeName:}" failed. No retries permitted until 2026-07-09 11:12:42.055173892 +0000 UTC m=+45.646905039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls") pod "dns-default-fmjjv" (UID: "49b90316-83e9-4bc1-a9b3-e118c434d9fd") : secret "dns-default-metrics-tls" not found
Jul 09 11:12:34.254847 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:34.254805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rzgk" event={"ID":"f1907393-916d-43d8-8ec1-0cb8029ad259","Type":"ContainerStarted","Data":"2954ed879e1d0ed11e0bdb7f0c56eb5a7894cdb668b8f7624013a68bee6afe1c"}
Jul 09 11:12:34.289038 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:34.288975    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8rzgk" podStartSLOduration=7.407657325 podStartE2EDuration="37.288956611s" totalImagesPullingTime="29.881299286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 11:11:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:12:34.286867489 +0000 UTC m=+37.878598654" watchObservedRunningTime="2026-07-09 11:12:34.288956611 +0000 UTC m=+37.880687775"
Jul 09 11:12:34.456526 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:34.456380    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:34.583623 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:34.583587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:34.680083 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:34.680053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:35.259377 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:35.259336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bzxtz" event={"ID":"e3650c56-8363-4f9a-94ec-e5de7bbea057","Type":"ContainerStarted","Data":"2c2ae33305826dd4265e1e65cf0ca9ab159a2ea4f257b9dcf6620a4ab084ba5e"}
Jul 09 11:12:35.259848 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:35.259565    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:12:35.279751 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:35.279690    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bzxtz" podStartSLOduration=34.809972139 podStartE2EDuration="38.279676729s" totalImagesPullingTime="3.46970459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:12:35.278406825 +0000 UTC m=+38.870137989" watchObservedRunningTime="2026-07-09 11:12:35.279676729 +0000 UTC m=+38.871407893"
Jul 09 11:12:35.773297 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:35.773246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:35.941607 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:35.941556    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:36.088393 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:36.088300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:36.188922 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:36.188882    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:37.265861 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:37.265823    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q7c8l" event={"ID":"346f1026-54ad-4ae6-9134-e2d1a31e565f","Type":"ContainerStarted","Data":"376213b56a95bbac60e76ad9ac6eec0b89829b5a0b66f3d163c45b2fe11129cb"}
Jul 09 11:12:42.111236 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:42.111188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:42.111734 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:42.111357    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 11:12:42.111734 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:42.111445    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls podName:49b90316-83e9-4bc1-a9b3-e118c434d9fd nodeName:}" failed. No retries permitted until 2026-07-09 11:12:58.111424162 +0000 UTC m=+61.703155304 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls") pod "dns-default-fmjjv" (UID: "49b90316-83e9-4bc1-a9b3-e118c434d9fd") : secret "dns-default-metrics-tls" not found
Jul 09 11:12:52.220834 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:52.220809    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8x75w"
Jul 09 11:12:52.253448 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:52.253389    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q7c8l" podStartSLOduration=49.927804902 podStartE2EDuration="55.253375439s" totalImagesPullingTime="5.325570537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 11:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:12:37.289190313 +0000 UTC m=+40.880921478" watchObservedRunningTime="2026-07-09 11:12:52.253375439 +0000 UTC m=+55.845106614"
Jul 09 11:12:52.364084 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:52.364047    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:52.464262 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:52.464227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:12:58.117507 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:12:58.117472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:12:58.117995 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:58.117600    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 11:12:58.117995 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:12:58.117653    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls podName:49b90316-83e9-4bc1-a9b3-e118c434d9fd nodeName:}" failed. No retries permitted until 2026-07-09 11:13:30.117637848 +0000 UTC m=+93.709368990 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls") pod "dns-default-fmjjv" (UID: "49b90316-83e9-4bc1-a9b3-e118c434d9fd") : secret "dns-default-metrics-tls" not found
Jul 09 11:13:01.640866 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:01.640822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs\") pod \"network-metrics-daemon-hphjc\" (UID: \"314935e3-3e91-4cc8-b5fc-15795b54d57a\") " pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:13:01.641281 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:13:01.640947    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 11:13:01.641281 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:13:01.641001    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs podName:314935e3-3e91-4cc8-b5fc-15795b54d57a nodeName:}" failed. No retries permitted until 2026-07-09 11:14:05.640987369 +0000 UTC m=+129.232718513 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs") pod "network-metrics-daemon-hphjc" (UID: "314935e3-3e91-4cc8-b5fc-15795b54d57a") : secret "metrics-daemon-secret" not found
Jul 09 11:13:06.264562 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:06.264532    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bzxtz"
Jul 09 11:13:10.894532 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.894497    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md"]
Jul 09 11:13:10.897373 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.897356    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md"
Jul 09 11:13:10.901298 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.901273    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 11:13:10.902319 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.902239    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 09 11:13:10.902319 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.902272    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 11:13:10.902319 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.902302    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 11:13:10.902576 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.902560    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5k92c\""
Jul 09 11:13:10.912076 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.912055    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md"]
Jul 09 11:13:10.957826 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.957803    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"]
Jul 09 11:13:10.959565 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.959550    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:10.962015 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.961988    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 09 11:13:10.962129 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.962086    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 09 11:13:10.962179 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.962137    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 09 11:13:10.962258 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.962244    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 09 11:13:10.978257 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.978232    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"]
Jul 09 11:13:11.000344 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:10.999987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3ea1046-12fc-4d3d-ae46-9e4a47a13581-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cc86c7578-qs4md\" (UID: \"e3ea1046-12fc-4d3d-ae46-9e4a47a13581\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md"
Jul 09 11:13:11.000344 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.000055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lzd7\" (UniqueName: \"kubernetes.io/projected/e3ea1046-12fc-4d3d-ae46-9e4a47a13581-kube-api-access-6lzd7\") pod \"managed-serviceaccount-addon-agent-5cc86c7578-qs4md\" (UID: \"e3ea1046-12fc-4d3d-ae46-9e4a47a13581\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md"
Jul 09 11:13:11.000344 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.000138    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/64f6df1d-0636-4395-8c10-023950c23414-ca\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.000344 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.000168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/64f6df1d-0636-4395-8c10-023950c23414-hub\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.000344 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.000214    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/64f6df1d-0636-4395-8c10-023950c23414-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.000344 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.000264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64f6df1d-0636-4395-8c10-023950c23414-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.000729 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.000400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f6df1d-0636-4395-8c10-023950c23414-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.000729 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.000478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r75lh\" (UniqueName: \"kubernetes.io/projected/64f6df1d-0636-4395-8c10-023950c23414-kube-api-access-r75lh\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.101230 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.101192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3ea1046-12fc-4d3d-ae46-9e4a47a13581-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cc86c7578-qs4md\" (UID: \"e3ea1046-12fc-4d3d-ae46-9e4a47a13581\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md"
Jul 09 11:13:11.101230 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.101233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lzd7\" (UniqueName: \"kubernetes.io/projected/e3ea1046-12fc-4d3d-ae46-9e4a47a13581-kube-api-access-6lzd7\") pod \"managed-serviceaccount-addon-agent-5cc86c7578-qs4md\" (UID: \"e3ea1046-12fc-4d3d-ae46-9e4a47a13581\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md"
Jul 09 11:13:11.101468 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.101283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/64f6df1d-0636-4395-8c10-023950c23414-ca\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.101468 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.101304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/64f6df1d-0636-4395-8c10-023950c23414-hub\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.101468 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.101325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/64f6df1d-0636-4395-8c10-023950c23414-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.101468 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.101350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64f6df1d-0636-4395-8c10-023950c23414-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.101468 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.101396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f6df1d-0636-4395-8c10-023950c23414-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.101468 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.101419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r75lh\" (UniqueName: \"kubernetes.io/projected/64f6df1d-0636-4395-8c10-023950c23414-kube-api-access-r75lh\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.102174 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.102147    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/64f6df1d-0636-4395-8c10-023950c23414-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.103947 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.103919    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3ea1046-12fc-4d3d-ae46-9e4a47a13581-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cc86c7578-qs4md\" (UID: \"e3ea1046-12fc-4d3d-ae46-9e4a47a13581\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md"
Jul 09 11:13:11.104151 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.104130    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/64f6df1d-0636-4395-8c10-023950c23414-ca\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.104397 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.104373    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/64f6df1d-0636-4395-8c10-023950c23414-hub\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.104397 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.104387    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/64f6df1d-0636-4395-8c10-023950c23414-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.104494 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.104392    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/64f6df1d-0636-4395-8c10-023950c23414-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.110415 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.110375    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lzd7\" (UniqueName: \"kubernetes.io/projected/e3ea1046-12fc-4d3d-ae46-9e4a47a13581-kube-api-access-6lzd7\") pod \"managed-serviceaccount-addon-agent-5cc86c7578-qs4md\" (UID: \"e3ea1046-12fc-4d3d-ae46-9e4a47a13581\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md"
Jul 09 11:13:11.110973 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.110955    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r75lh\" (UniqueName: \"kubernetes.io/projected/64f6df1d-0636-4395-8c10-023950c23414-kube-api-access-r75lh\") pod \"cluster-proxy-proxy-agent-6b54648947-c24rx\" (UID: \"64f6df1d-0636-4395-8c10-023950c23414\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.218713 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.218618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md"
Jul 09 11:13:11.267936 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.267905    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:13:11.369062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.369035    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md"]
Jul 09 11:13:11.430572 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:11.430539    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"]
Jul 09 11:13:11.433655 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:13:11.433630    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64f6df1d_0636_4395_8c10_023950c23414.slice/crio-17d0fb01fdb83cd308044402355ff451ea9a242f940ce5d5601e280c91afc481 WatchSource:0}: Error finding container 17d0fb01fdb83cd308044402355ff451ea9a242f940ce5d5601e280c91afc481: Status 404 returned error can't find the container with id 17d0fb01fdb83cd308044402355ff451ea9a242f940ce5d5601e280c91afc481
Jul 09 11:13:12.350676 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:12.350632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx" event={"ID":"64f6df1d-0636-4395-8c10-023950c23414","Type":"ContainerStarted","Data":"17d0fb01fdb83cd308044402355ff451ea9a242f940ce5d5601e280c91afc481"}
Jul 09 11:13:12.352799 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:12.352756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md" event={"ID":"e3ea1046-12fc-4d3d-ae46-9e4a47a13581","Type":"ContainerStarted","Data":"3ad114504d68dd279bb8bd33dca9bfb399bbaaefaf714bc318859150e2a5aa71"}
Jul 09 11:13:14.361270 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:14.361219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md" event={"ID":"e3ea1046-12fc-4d3d-ae46-9e4a47a13581","Type":"ContainerStarted","Data":"4fca7eda7e0f19350e22acbefc392f2f246fb117f4094b3ac6c6d86f7bfb0886"}
Jul 09 11:13:14.381403 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:14.381336    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md" podStartSLOduration=1.918014104 podStartE2EDuration="4.381315794s" totalImagesPullingTime="2.46330169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:13:14.381299204 +0000 UTC m=+77.973030368" watchObservedRunningTime="2026-07-09 11:13:14.381315794 +0000 UTC m=+77.973046960"
Jul 09 11:13:15.369647 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:15.369553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx" event={"ID":"64f6df1d-0636-4395-8c10-023950c23414","Type":"ContainerStarted","Data":"60439454af8291ee916c608aa65b397b92fdd9b0371b31e7a9e25fd454c7b2eb"}
Jul 09 11:13:15.369647 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:15.369595    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx" event={"ID":"64f6df1d-0636-4395-8c10-023950c23414","Type":"ContainerStarted","Data":"ac77050c41c99a58b02ef5bbb8068b87dec9a4a35bf32f38b848a75fd1ec701b"}
Jul 09 11:13:15.369647 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:15.369609    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx" event={"ID":"64f6df1d-0636-4395-8c10-023950c23414","Type":"ContainerStarted","Data":"3a0caa80c50d137a53641fff21e9ad29292d8c04a631e6b7775a7bd534a71e96"}
Jul 09 11:13:15.392468 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:15.392411    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx" podStartSLOduration=1.706802953 podStartE2EDuration="5.392395388s" totalImagesPullingTime="3.685592435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:13:15.391804906 +0000 UTC m=+78.983536066" watchObservedRunningTime="2026-07-09 11:13:15.392395388 +0000 UTC m=+78.984126552"
Jul 09 11:13:30.141470 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:13:30.141337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:13:30.141470 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:13:30.141458    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 11:13:30.142004 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:13:30.141511    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls podName:49b90316-83e9-4bc1-a9b3-e118c434d9fd nodeName:}" failed. No retries permitted until 2026-07-09 11:14:34.141496298 +0000 UTC m=+157.733227440 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls") pod "dns-default-fmjjv" (UID: "49b90316-83e9-4bc1-a9b3-e118c434d9fd") : secret "dns-default-metrics-tls" not found
Jul 09 11:14:05.694470 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:14:05.694433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs\") pod \"network-metrics-daemon-hphjc\" (UID: \"314935e3-3e91-4cc8-b5fc-15795b54d57a\") " pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:14:05.694983 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:14:05.694578    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 11:14:05.694983 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:14:05.694650    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs podName:314935e3-3e91-4cc8-b5fc-15795b54d57a nodeName:}" failed. No retries permitted until 2026-07-09 11:16:07.694635035 +0000 UTC m=+251.286366177 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs") pod "network-metrics-daemon-hphjc" (UID: "314935e3-3e91-4cc8-b5fc-15795b54d57a") : secret "metrics-daemon-secret" not found
Jul 09 11:14:29.347228 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:14:29.347170    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fmjjv" podUID="49b90316-83e9-4bc1-a9b3-e118c434d9fd"
Jul 09 11:14:29.573903 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:14:29.573871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:14:29.958969 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:14:29.958926    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hphjc" podUID="314935e3-3e91-4cc8-b5fc-15795b54d57a"
Jul 09 11:14:34.189292 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:14:34.189235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:14:34.189686 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:14:34.189384    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 11:14:34.189686 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:14:34.189453    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls podName:49b90316-83e9-4bc1-a9b3-e118c434d9fd nodeName:}" failed. No retries permitted until 2026-07-09 11:16:36.189437419 +0000 UTC m=+279.781168561 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls") pod "dns-default-fmjjv" (UID: "49b90316-83e9-4bc1-a9b3-e118c434d9fd") : secret "dns-default-metrics-tls" not found
Jul 09 11:14:41.944528 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:14:41.944490    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:15:12.988189 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:12.988155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:13.706238 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:13.706203    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:14.701749 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:14.701715    2563 generic.go:356] "Generic (PLEG): container finished" podID="e3ea1046-12fc-4d3d-ae46-9e4a47a13581" containerID="4fca7eda7e0f19350e22acbefc392f2f246fb117f4094b3ac6c6d86f7bfb0886" exitCode=255
Jul 09 11:15:14.702151 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:14.701766    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md" event={"ID":"e3ea1046-12fc-4d3d-ae46-9e4a47a13581","Type":"ContainerDied","Data":"4fca7eda7e0f19350e22acbefc392f2f246fb117f4094b3ac6c6d86f7bfb0886"}
Jul 09 11:15:14.702151 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:14.702041    2563 scope.go:117] "RemoveContainer" containerID="4fca7eda7e0f19350e22acbefc392f2f246fb117f4094b3ac6c6d86f7bfb0886"
Jul 09 11:15:15.709765 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:15.709730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cc86c7578-qs4md" event={"ID":"e3ea1046-12fc-4d3d-ae46-9e4a47a13581","Type":"ContainerStarted","Data":"68541e81f3957f300a263d214bd7903652859ee526a59392702617d127bc8df4"}
Jul 09 11:15:24.944093 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:24.944062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:49.573777 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:49.573742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:49.666411 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:49.666378    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:52.230668 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:52.230639    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:52.308011 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:52.307980    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:53.302075 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.302042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:53.500849 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.500816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:53.541633 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.541603    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gmlwh"]
Jul 09 11:15:53.544335 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.544310    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:53.547132 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.547110    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 11:15:53.548245 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.548222    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 11:15:53.548369 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.548221    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8hzfb\""
Jul 09 11:15:53.548369 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.548275    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 11:15:53.548369 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.548224    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 11:15:53.559011 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.558947    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gmlwh"]
Jul 09 11:15:53.613223 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.613182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:53.618188 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.618163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbwgm\" (UniqueName: \"kubernetes.io/projected/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-api-access-jbwgm\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:53.618333 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.618209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5569f56e-a640-4762-b216-9dbd9e18f3f5-crio-socket\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:53.618333 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.618272    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:53.618333 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.618308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:53.618333 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.618326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5569f56e-a640-4762-b216-9dbd9e18f3f5-data-volume\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:53.718823 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.718768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:53.718962 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.718857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:53.718962 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.718887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5569f56e-a640-4762-b216-9dbd9e18f3f5-data-volume\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:53.718962 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.718924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbwgm\" (UniqueName: \"kubernetes.io/projected/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-api-access-jbwgm\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:53.718962 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:53.718941    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 11:15:53.718962 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.718960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5569f56e-a640-4762-b216-9dbd9e18f3f5-crio-socket\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:53.719185 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:53.718986    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 11:15:53.719185 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:53.719019    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls podName:5569f56e-a640-4762-b216-9dbd9e18f3f5 nodeName:}" failed. No retries permitted until 2026-07-09 11:15:54.218999947 +0000 UTC m=+237.810731096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gmlwh" (UID: "5569f56e-a640-4762-b216-9dbd9e18f3f5") : secret "insights-runtime-extractor-tls" not found
Jul 09 11:15:53.719185 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.719053    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5569f56e-a640-4762-b216-9dbd9e18f3f5-crio-socket\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:53.719185 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:53.719071    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm podName:5569f56e-a640-4762-b216-9dbd9e18f3f5 nodeName:}" failed. No retries permitted until 2026-07-09 11:15:54.219052378 +0000 UTC m=+237.810783521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gmlwh" (UID: "5569f56e-a640-4762-b216-9dbd9e18f3f5") : configmap "kube-rbac-proxy" not found
Jul 09 11:15:53.719375 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.719242    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5569f56e-a640-4762-b216-9dbd9e18f3f5-data-volume\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:53.729239 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.729207    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbwgm\" (UniqueName: \"kubernetes.io/projected/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-api-access-jbwgm\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:53.903830 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.903728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:53.999904 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:53.999870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:54.222574 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:54.222460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:54.222574 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:54.222536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:54.222859 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:54.222625    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 11:15:54.222859 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:54.222708    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls podName:5569f56e-a640-4762-b216-9dbd9e18f3f5 nodeName:}" failed. No retries permitted until 2026-07-09 11:15:55.222691201 +0000 UTC m=+238.814422344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gmlwh" (UID: "5569f56e-a640-4762-b216-9dbd9e18f3f5") : secret "insights-runtime-extractor-tls" not found
Jul 09 11:15:54.222859 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:54.222628    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 11:15:54.222859 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:54.222818    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm podName:5569f56e-a640-4762-b216-9dbd9e18f3f5 nodeName:}" failed. No retries permitted until 2026-07-09 11:15:55.222776825 +0000 UTC m=+238.814507972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gmlwh" (UID: "5569f56e-a640-4762-b216-9dbd9e18f3f5") : configmap "kube-rbac-proxy" not found
Jul 09 11:15:54.341336 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:54.341304    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:54.415870 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:54.415838    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:54.673458 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:54.673425    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:54.800753 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:54.800723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:55.230928 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:55.230880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:55.230928 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:55.230942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:55.231191 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:55.231011    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 11:15:55.231191 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:55.231041    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 11:15:55.231191 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:55.231089    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm podName:5569f56e-a640-4762-b216-9dbd9e18f3f5 nodeName:}" failed. No retries permitted until 2026-07-09 11:15:57.231072891 +0000 UTC m=+240.822804032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gmlwh" (UID: "5569f56e-a640-4762-b216-9dbd9e18f3f5") : configmap "kube-rbac-proxy" not found
Jul 09 11:15:55.231191 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:55.231104    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls podName:5569f56e-a640-4762-b216-9dbd9e18f3f5 nodeName:}" failed. No retries permitted until 2026-07-09 11:15:57.231097889 +0000 UTC m=+240.822829031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gmlwh" (UID: "5569f56e-a640-4762-b216-9dbd9e18f3f5") : secret "insights-runtime-extractor-tls" not found
Jul 09 11:15:56.236703 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.236667    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-pndg8"]
Jul 09 11:15:56.239754 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.239731    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-pndg8"
Jul 09 11:15:56.242682 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.242664    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 09 11:15:56.242956 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.242941    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-k6sz9\""
Jul 09 11:15:56.243794 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.243769    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 09 11:15:56.243866 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.243797    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 09 11:15:56.243926 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.243799    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 09 11:15:56.255200 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.255183    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-pndg8"]
Jul 09 11:15:56.261802 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.261765    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:56.291580 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.291554    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k7d7z_72b5c8e9-cd8b-4dde-9850-824fbf0818e7/dns-node-resolver/0.log"
Jul 09 11:15:56.339194 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.339154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/77eb51b9-2397-428e-8980-3852d113ba85-signing-key\") pod \"service-ca-758f599c5-pndg8\" (UID: \"77eb51b9-2397-428e-8980-3852d113ba85\") " pod="openshift-service-ca/service-ca-758f599c5-pndg8"
Jul 09 11:15:56.339333 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.339205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/77eb51b9-2397-428e-8980-3852d113ba85-signing-cabundle\") pod \"service-ca-758f599c5-pndg8\" (UID: \"77eb51b9-2397-428e-8980-3852d113ba85\") " pod="openshift-service-ca/service-ca-758f599c5-pndg8"
Jul 09 11:15:56.339333 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.339303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2kr7\" (UniqueName: \"kubernetes.io/projected/77eb51b9-2397-428e-8980-3852d113ba85-kube-api-access-z2kr7\") pod \"service-ca-758f599c5-pndg8\" (UID: \"77eb51b9-2397-428e-8980-3852d113ba85\") " pod="openshift-service-ca/service-ca-758f599c5-pndg8"
Jul 09 11:15:56.359516 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.359490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:56.440471 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.440438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/77eb51b9-2397-428e-8980-3852d113ba85-signing-cabundle\") pod \"service-ca-758f599c5-pndg8\" (UID: \"77eb51b9-2397-428e-8980-3852d113ba85\") " pod="openshift-service-ca/service-ca-758f599c5-pndg8"
Jul 09 11:15:56.440577 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.440509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2kr7\" (UniqueName: \"kubernetes.io/projected/77eb51b9-2397-428e-8980-3852d113ba85-kube-api-access-z2kr7\") pod \"service-ca-758f599c5-pndg8\" (UID: \"77eb51b9-2397-428e-8980-3852d113ba85\") " pod="openshift-service-ca/service-ca-758f599c5-pndg8"
Jul 09 11:15:56.440577 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.440551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/77eb51b9-2397-428e-8980-3852d113ba85-signing-key\") pod \"service-ca-758f599c5-pndg8\" (UID: \"77eb51b9-2397-428e-8980-3852d113ba85\") " pod="openshift-service-ca/service-ca-758f599c5-pndg8"
Jul 09 11:15:56.441198 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.441179    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/77eb51b9-2397-428e-8980-3852d113ba85-signing-cabundle\") pod \"service-ca-758f599c5-pndg8\" (UID: \"77eb51b9-2397-428e-8980-3852d113ba85\") " pod="openshift-service-ca/service-ca-758f599c5-pndg8"
Jul 09 11:15:56.442969 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.442949    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/77eb51b9-2397-428e-8980-3852d113ba85-signing-key\") pod \"service-ca-758f599c5-pndg8\" (UID: \"77eb51b9-2397-428e-8980-3852d113ba85\") " pod="openshift-service-ca/service-ca-758f599c5-pndg8"
Jul 09 11:15:56.454041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.454017    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2kr7\" (UniqueName: \"kubernetes.io/projected/77eb51b9-2397-428e-8980-3852d113ba85-kube-api-access-z2kr7\") pod \"service-ca-758f599c5-pndg8\" (UID: \"77eb51b9-2397-428e-8980-3852d113ba85\") " pod="openshift-service-ca/service-ca-758f599c5-pndg8"
Jul 09 11:15:56.548636 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.548604    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-pndg8"
Jul 09 11:15:56.676504 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.676473    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-pndg8"]
Jul 09 11:15:56.679487 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:15:56.679450    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77eb51b9_2397_428e_8980_3852d113ba85.slice/crio-04d37f0ead69e6fb6b9beeb264364f70ca50cb0246f754d60c12326dc2a2c08f WatchSource:0}: Error finding container 04d37f0ead69e6fb6b9beeb264364f70ca50cb0246f754d60c12326dc2a2c08f: Status 404 returned error can't find the container with id 04d37f0ead69e6fb6b9beeb264364f70ca50cb0246f754d60c12326dc2a2c08f
Jul 09 11:15:56.831262 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:56.831167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-pndg8" event={"ID":"77eb51b9-2397-428e-8980-3852d113ba85","Type":"ContainerStarted","Data":"04d37f0ead69e6fb6b9beeb264364f70ca50cb0246f754d60c12326dc2a2c08f"}
Jul 09 11:15:57.248271 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:57.248180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:57.248729 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:57.248294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:15:57.248729 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:57.248335    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 11:15:57.248729 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:57.248420    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm podName:5569f56e-a640-4762-b216-9dbd9e18f3f5 nodeName:}" failed. No retries permitted until 2026-07-09 11:16:01.248398906 +0000 UTC m=+244.840130068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gmlwh" (UID: "5569f56e-a640-4762-b216-9dbd9e18f3f5") : configmap "kube-rbac-proxy" not found
Jul 09 11:15:57.248729 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:57.248435    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 11:15:57.248729 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:15:57.248487    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls podName:5569f56e-a640-4762-b216-9dbd9e18f3f5 nodeName:}" failed. No retries permitted until 2026-07-09 11:16:01.248471546 +0000 UTC m=+244.840202690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gmlwh" (UID: "5569f56e-a640-4762-b216-9dbd9e18f3f5") : secret "insights-runtime-extractor-tls" not found
Jul 09 11:15:57.491429 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:57.491391    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tx572_00d1d258-b95a-4f9e-88ac-ac35af980a14/node-ca/0.log"
Jul 09 11:15:57.893305 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:57.893275    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-745gr_53a84cad-5a9e-4284-95f4-7ed61d11ac64/serve-healthcheck-canary/0.log"
Jul 09 11:15:58.206955 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:58.206881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:58.838249 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:58.838210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-pndg8" event={"ID":"77eb51b9-2397-428e-8980-3852d113ba85","Type":"ContainerStarted","Data":"b27c067b69594445f6a58dbe4b187b67b42c0f1e519112687f6532baaff17b69"}
Jul 09 11:15:58.888989 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:58.888929    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-pndg8" podStartSLOduration=0.850315758 podStartE2EDuration="2.888906978s" totalImagesPullingTime="2.03859122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:15:58.886804036 +0000 UTC m=+242.478535191" watchObservedRunningTime="2026-07-09 11:15:58.888906978 +0000 UTC m=+242.480638143"
Jul 09 11:15:59.615604 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:59.615567    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:59.726160 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:59.726126    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:59.833904 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:59.833862    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:15:59.974900 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:15:59.974809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:00.143434 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:00.143377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:00.417202 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:00.417169    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:01.281179 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:01.281151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:16:01.281558 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:01.281197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:16:01.281558 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:16:01.281305    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 11:16:01.281558 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:16:01.281345    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 11:16:01.281558 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:16:01.281367    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls podName:5569f56e-a640-4762-b216-9dbd9e18f3f5 nodeName:}" failed. No retries permitted until 2026-07-09 11:16:09.281350358 +0000 UTC m=+252.873081513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gmlwh" (UID: "5569f56e-a640-4762-b216-9dbd9e18f3f5") : secret "insights-runtime-extractor-tls" not found
Jul 09 11:16:01.281558 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:16:01.281385    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm podName:5569f56e-a640-4762-b216-9dbd9e18f3f5 nodeName:}" failed. No retries permitted until 2026-07-09 11:16:09.28137383 +0000 UTC m=+252.873104973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gmlwh" (UID: "5569f56e-a640-4762-b216-9dbd9e18f3f5") : configmap "kube-rbac-proxy" not found
Jul 09 11:16:01.585208 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:01.585120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:01.810503 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:01.810471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:01.971063 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:01.970975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:02.173042 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:02.173006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:02.345193 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:02.345161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:03.360270 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:03.360235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:03.572368 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:03.572326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:03.726225 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:03.726140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:03.877765 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:03.877731    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:04.023767 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:04.023740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:04.143171 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:04.143140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:04.342937 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:04.342847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:04.476801 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:04.476753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:05.569278 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:05.569238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:05.747425 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:05.747393    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:05.957570 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:05.957484    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:06.212054 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:06.211965    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:06.410864 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:06.410834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:07.137981 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:07.137949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:07.728212 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:07.728183    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:07.733239 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:07.733219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs\") pod \"network-metrics-daemon-hphjc\" (UID: \"314935e3-3e91-4cc8-b5fc-15795b54d57a\") " pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:16:07.735569 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:07.735544    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/314935e3-3e91-4cc8-b5fc-15795b54d57a-metrics-certs\") pod \"network-metrics-daemon-hphjc\" (UID: \"314935e3-3e91-4cc8-b5fc-15795b54d57a\") " pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:16:07.748471 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:07.748453    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h6b4g\""
Jul 09 11:16:07.755928 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:07.755913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphjc"
Jul 09 11:16:07.902883 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:07.902698    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hphjc"]
Jul 09 11:16:07.905408 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:16:07.905380    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314935e3_3e91_4cc8_b5fc_15795b54d57a.slice/crio-f462e4e088c286194c1ec6a2262c26c03440817725ccda80699f0dd9ff8dd8ed WatchSource:0}: Error finding container f462e4e088c286194c1ec6a2262c26c03440817725ccda80699f0dd9ff8dd8ed: Status 404 returned error can't find the container with id f462e4e088c286194c1ec6a2262c26c03440817725ccda80699f0dd9ff8dd8ed
Jul 09 11:16:07.951082 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:07.951049    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:08.383685 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:08.383643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:08.680592 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:08.680512    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:08.869054 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:08.869017    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hphjc" event={"ID":"314935e3-3e91-4cc8-b5fc-15795b54d57a","Type":"ContainerStarted","Data":"f462e4e088c286194c1ec6a2262c26c03440817725ccda80699f0dd9ff8dd8ed"}
Jul 09 11:16:08.970063 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:08.969983    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:09.348026 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:09.347985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:16:09.348217 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:09.348043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:16:09.348217 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:16:09.348157    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 11:16:09.348310 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:16:09.348232    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls podName:5569f56e-a640-4762-b216-9dbd9e18f3f5 nodeName:}" failed. No retries permitted until 2026-07-09 11:16:25.348211244 +0000 UTC m=+268.939942396 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gmlwh" (UID: "5569f56e-a640-4762-b216-9dbd9e18f3f5") : secret "insights-runtime-extractor-tls" not found
Jul 09 11:16:09.348679 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:09.348660    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5569f56e-a640-4762-b216-9dbd9e18f3f5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:16:09.529600 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:09.529567    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:09.874054 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:09.873964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hphjc" event={"ID":"314935e3-3e91-4cc8-b5fc-15795b54d57a","Type":"ContainerStarted","Data":"09f4fc7faa06a97f043f28f4c1b9d81d5a0455790af7d2514a95b10d70270de0"}
Jul 09 11:16:09.874054 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:09.874045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hphjc" event={"ID":"314935e3-3e91-4cc8-b5fc-15795b54d57a","Type":"ContainerStarted","Data":"56511f0f5c7ffdfb9b11a8b8d0a5101e72cfe3f92cc7fc9cf7da50ae944738bb"}
Jul 09 11:16:09.900446 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:09.900393    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hphjc" podStartSLOduration=252.678323958 podStartE2EDuration="4m13.900380261s" totalImagesPullingTime="1.222056303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:16:09.900337457 +0000 UTC m=+253.492068623" watchObservedRunningTime="2026-07-09 11:16:09.900380261 +0000 UTC m=+253.492111424"
Jul 09 11:16:09.930534 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:09.930503    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:10.121371 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:10.121339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:10.318147 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:10.318116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:12.444565 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:12.444533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:12.529837 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:12.529812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:13.253494 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:13.253463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:13.434035 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:13.434001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:20.021191 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:20.021108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:20.139412 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:20.139360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:20.793112 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:20.793064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:20.942142 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:20.942107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:22.811862 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:22.811831    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:23.034382 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:23.034339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:24.417037 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:24.417006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:24.564526 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:24.564493    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:24.834114 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:24.834086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:24.979117 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:24.979083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:25.113614 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:25.113534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:25.363295 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:25.363257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:16:25.365668 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:25.365617    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5569f56e-a640-4762-b216-9dbd9e18f3f5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gmlwh\" (UID: \"5569f56e-a640-4762-b216-9dbd9e18f3f5\") " pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:16:25.450944 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:25.450911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:25.657989 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:25.657911    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8hzfb\""
Jul 09 11:16:25.664225 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:25.664209    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gmlwh"
Jul 09 11:16:25.813633 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:25.813607    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gmlwh"]
Jul 09 11:16:25.816138 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:16:25.816088    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5569f56e_a640_4762_b216_9dbd9e18f3f5.slice/crio-ef5749d5ce57a4e38c65e59f767f2173cff0f3e4717059cbc8ab6ab958dc6c88 WatchSource:0}: Error finding container ef5749d5ce57a4e38c65e59f767f2173cff0f3e4717059cbc8ab6ab958dc6c88: Status 404 returned error can't find the container with id ef5749d5ce57a4e38c65e59f767f2173cff0f3e4717059cbc8ab6ab958dc6c88
Jul 09 11:16:25.923267 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:25.923186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gmlwh" event={"ID":"5569f56e-a640-4762-b216-9dbd9e18f3f5","Type":"ContainerStarted","Data":"ef5749d5ce57a4e38c65e59f767f2173cff0f3e4717059cbc8ab6ab958dc6c88"}
Jul 09 11:16:27.996827 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:27.996768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:28.161460 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:28.161410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:28.264597 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:28.264562    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:29.938824 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:29.938730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gmlwh" event={"ID":"5569f56e-a640-4762-b216-9dbd9e18f3f5","Type":"ContainerStarted","Data":"95379a18986a17b20b4022b93f4fca7353aada220ad0489c5a6019dd25fd7991"}
Jul 09 11:16:30.943151 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:30.943118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gmlwh" event={"ID":"5569f56e-a640-4762-b216-9dbd9e18f3f5","Type":"ContainerStarted","Data":"88aeff1610d76a3610f0fb7e98b139c5febeec55309aff2684bcd0cc127e21ee"}
Jul 09 11:16:31.357913 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:31.357874    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:32.575404 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:16:32.575357    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fmjjv" podUID="49b90316-83e9-4bc1-a9b3-e118c434d9fd"
Jul 09 11:16:32.948869 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:32.948801    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:16:33.328273 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:33.328242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:33.632587 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:33.632509    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:33.955733 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:33.955650    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gmlwh" event={"ID":"5569f56e-a640-4762-b216-9dbd9e18f3f5","Type":"ContainerStarted","Data":"790bc812d471c790a4a4cc989dfd8f3e570bcae010b5622f1e8ad52989f58892"}
Jul 09 11:16:33.979647 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:33.979596    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gmlwh" podStartSLOduration=33.941827472 podStartE2EDuration="40.979581041s" totalImagesPullingTime="7.037753569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 11:15:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:16:33.978637097 +0000 UTC m=+277.570368261" watchObservedRunningTime="2026-07-09 11:16:33.979581041 +0000 UTC m=+277.571312206"
Jul 09 11:16:36.245627 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:36.245595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:16:36.247826 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:36.247808    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49b90316-83e9-4bc1-a9b3-e118c434d9fd-metrics-tls\") pod \"dns-default-fmjjv\" (UID: \"49b90316-83e9-4bc1-a9b3-e118c434d9fd\") " pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:16:36.253081 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:36.253057    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6pmqk\""
Jul 09 11:16:36.260084 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:36.260066    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:16:36.393546 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:16:36.393516    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b90316_83e9_4bc1_a9b3_e118c434d9fd.slice/crio-5602aac087aa2136233a795ca1e70602a2ffec57d858dd2712f3d9a4a794ebde WatchSource:0}: Error finding container 5602aac087aa2136233a795ca1e70602a2ffec57d858dd2712f3d9a4a794ebde: Status 404 returned error can't find the container with id 5602aac087aa2136233a795ca1e70602a2ffec57d858dd2712f3d9a4a794ebde
Jul 09 11:16:36.393964 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:36.393942    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fmjjv"]
Jul 09 11:16:36.965088 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:36.965049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fmjjv" event={"ID":"49b90316-83e9-4bc1-a9b3-e118c434d9fd","Type":"ContainerStarted","Data":"5602aac087aa2136233a795ca1e70602a2ffec57d858dd2712f3d9a4a794ebde"}
Jul 09 11:16:37.254184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.254145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:37.379630 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.379597    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8pslv"]
Jul 09 11:16:37.382352 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.382325    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.385545 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:16:37.385511    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"node-exporter-accelerators-collector-config\" is forbidden: User \"system:node:ip-10-0-138-176.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-138-176.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap"
Jul 09 11:16:37.386284 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.386260    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 11:16:37.386398 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.386346    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 11:16:37.386900 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.386875    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 11:16:37.387492 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.387471    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 11:16:37.387687 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.387673    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kn2d9\""
Jul 09 11:16:37.388045 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.387899    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 11:16:37.454055 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.454020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8176269c-37d1-47ed-9d2d-a7c49244ebe4-root\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.454228 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.454098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.454228 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.454126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-accelerators-collector-config\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.454228 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.454155    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-textfile\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.454228 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.454196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8176269c-37d1-47ed-9d2d-a7c49244ebe4-metrics-client-ca\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.454228 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.454219    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8176269c-37d1-47ed-9d2d-a7c49244ebe4-sys\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.454467 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.454241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-tls\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.454467 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.454347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srpcb\" (UniqueName: \"kubernetes.io/projected/8176269c-37d1-47ed-9d2d-a7c49244ebe4-kube-api-access-srpcb\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.454467 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.454393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-wtmp\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.555103 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.554980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8176269c-37d1-47ed-9d2d-a7c49244ebe4-sys\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.555103 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.555033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-tls\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.555103 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.555069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srpcb\" (UniqueName: \"kubernetes.io/projected/8176269c-37d1-47ed-9d2d-a7c49244ebe4-kube-api-access-srpcb\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.555103 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.555099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-wtmp\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.555414 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.555110    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8176269c-37d1-47ed-9d2d-a7c49244ebe4-sys\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.555414 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.555129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8176269c-37d1-47ed-9d2d-a7c49244ebe4-root\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.555414 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.555173    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8176269c-37d1-47ed-9d2d-a7c49244ebe4-root\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.555414 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.555195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.555414 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.555234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-accelerators-collector-config\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.555414 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.555274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-textfile\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.555414 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.555295    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-wtmp\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.555414 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.555305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8176269c-37d1-47ed-9d2d-a7c49244ebe4-metrics-client-ca\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.555815 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.555612    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-textfile\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.555999 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.555976    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8176269c-37d1-47ed-9d2d-a7c49244ebe4-metrics-client-ca\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.557847 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.557803    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-tls\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.558020 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.557999    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:37.564717 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:37.564672    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srpcb\" (UniqueName: \"kubernetes.io/projected/8176269c-37d1-47ed-9d2d-a7c49244ebe4-kube-api-access-srpcb\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:38.555986 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:16:38.555946    2563 configmap.go:193] Couldn't get configMap openshift-monitoring/node-exporter-accelerators-collector-config: failed to sync configmap cache: timed out waiting for the condition
Jul 09 11:16:38.556421 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:16:38.556052    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-accelerators-collector-config podName:8176269c-37d1-47ed-9d2d-a7c49244ebe4 nodeName:}" failed. No retries permitted until 2026-07-09 11:16:39.056028104 +0000 UTC m=+282.647759246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-accelerators-collector-config" (UniqueName: "kubernetes.io/configmap/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-accelerators-collector-config") pod "node-exporter-8pslv" (UID: "8176269c-37d1-47ed-9d2d-a7c49244ebe4") : failed to sync configmap cache: timed out waiting for the condition
Jul 09 11:16:38.677218 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:38.677184    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 11:16:38.831637 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:38.831555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:38.925389 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:38.925360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:38.972130 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:38.972095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fmjjv" event={"ID":"49b90316-83e9-4bc1-a9b3-e118c434d9fd","Type":"ContainerStarted","Data":"20e9ca1ab4d7a90ae068ff450bfca3620165b88a46e7382dba9c1db92e6c4fba"}
Jul 09 11:16:38.972266 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:38.972146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fmjjv" event={"ID":"49b90316-83e9-4bc1-a9b3-e118c434d9fd","Type":"ContainerStarted","Data":"146d318d16d473f586c510bf03e205e3280f737caf2ec864313c0060659ef627"}
Jul 09 11:16:38.972266 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:38.972225    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:16:38.995034 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:38.994990    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fmjjv" podStartSLOduration=251.401713322 podStartE2EDuration="4m12.994974733s" totalImagesPullingTime="1.593261411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:16:38.992621999 +0000 UTC m=+282.584353164" watchObservedRunningTime="2026-07-09 11:16:38.994974733 +0000 UTC m=+282.586705887"
Jul 09 11:16:39.068208 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:39.068165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-accelerators-collector-config\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:39.068838 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:39.068821    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8176269c-37d1-47ed-9d2d-a7c49244ebe4-node-exporter-accelerators-collector-config\") pod \"node-exporter-8pslv\" (UID: \"8176269c-37d1-47ed-9d2d-a7c49244ebe4\") " pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:39.194618 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:39.194543    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8pslv"
Jul 09 11:16:39.204481 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:16:39.204453    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8176269c_37d1_47ed_9d2d_a7c49244ebe4.slice/crio-1517f59ab227678d02cdbcedc61c998a2eefae8489340bb6824d079aa800f2aa WatchSource:0}: Error finding container 1517f59ab227678d02cdbcedc61c998a2eefae8489340bb6824d079aa800f2aa: Status 404 returned error can't find the container with id 1517f59ab227678d02cdbcedc61c998a2eefae8489340bb6824d079aa800f2aa
Jul 09 11:16:39.980051 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:39.980018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8pslv" event={"ID":"8176269c-37d1-47ed-9d2d-a7c49244ebe4","Type":"ContainerStarted","Data":"1517f59ab227678d02cdbcedc61c998a2eefae8489340bb6824d079aa800f2aa"}
Jul 09 11:16:40.345668 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:40.345633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:40.454003 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:40.453969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:40.685576 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:40.685535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:40.984065 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:40.983979    2563 generic.go:356] "Generic (PLEG): container finished" podID="8176269c-37d1-47ed-9d2d-a7c49244ebe4" containerID="57374567eb5f68072debbaa286f8a651ccf2d9f3a3508dfb62b00ec6de958be9" exitCode=0
Jul 09 11:16:40.984065 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:40.984050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8pslv" event={"ID":"8176269c-37d1-47ed-9d2d-a7c49244ebe4","Type":"ContainerDied","Data":"57374567eb5f68072debbaa286f8a651ccf2d9f3a3508dfb62b00ec6de958be9"}
Jul 09 11:16:41.443211 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.443172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:41.703429 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.703343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:41.828475 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.828448    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-67b697bddd-26n9n"]
Jul 09 11:16:41.831435 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.831418    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.835410 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.835385    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 09 11:16:41.836111 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.836090    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6pksk\""
Jul 09 11:16:41.836741 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.836696    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3bekpcl6tu1b5\""
Jul 09 11:16:41.836741 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.836718    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 09 11:16:41.836912 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.836720    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 09 11:16:41.836912 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.836720    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 09 11:16:41.856463 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.854473    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67b697bddd-26n9n"]
Jul 09 11:16:41.886833 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.886812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:41.888872 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.888853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/801b8517-ff16-41b5-a455-24fd8e17d0d3-secret-metrics-server-tls\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.888945 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.888881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/801b8517-ff16-41b5-a455-24fd8e17d0d3-metrics-server-audit-profiles\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.888945 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.888909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/801b8517-ff16-41b5-a455-24fd8e17d0d3-secret-metrics-server-client-certs\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.889056 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.888977    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/801b8517-ff16-41b5-a455-24fd8e17d0d3-client-ca-bundle\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.889056 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.889041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/801b8517-ff16-41b5-a455-24fd8e17d0d3-audit-log\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.889133 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.889072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnwhd\" (UniqueName: \"kubernetes.io/projected/801b8517-ff16-41b5-a455-24fd8e17d0d3-kube-api-access-cnwhd\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.889133 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.889120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/801b8517-ff16-41b5-a455-24fd8e17d0d3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.989233 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.989144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8pslv" event={"ID":"8176269c-37d1-47ed-9d2d-a7c49244ebe4","Type":"ContainerStarted","Data":"ae36743c81feda6177d9567511db19d5084a1367d2e281467695183f7e518f4c"}
Jul 09 11:16:41.989233 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.989203    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8pslv" event={"ID":"8176269c-37d1-47ed-9d2d-a7c49244ebe4","Type":"ContainerStarted","Data":"9a7c265ce6acf1627fc29f08d9384f4c37dadd32c8e02f7450d1673f04ecb694"}
Jul 09 11:16:41.989746 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.989402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/801b8517-ff16-41b5-a455-24fd8e17d0d3-audit-log\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.989746 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.989430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnwhd\" (UniqueName: \"kubernetes.io/projected/801b8517-ff16-41b5-a455-24fd8e17d0d3-kube-api-access-cnwhd\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.989746 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.989461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/801b8517-ff16-41b5-a455-24fd8e17d0d3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.989746 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.989485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/801b8517-ff16-41b5-a455-24fd8e17d0d3-secret-metrics-server-tls\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.989746 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.989511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/801b8517-ff16-41b5-a455-24fd8e17d0d3-metrics-server-audit-profiles\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.989746 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.989547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/801b8517-ff16-41b5-a455-24fd8e17d0d3-secret-metrics-server-client-certs\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.989746 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.989573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/801b8517-ff16-41b5-a455-24fd8e17d0d3-client-ca-bundle\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.990111 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.989890    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/801b8517-ff16-41b5-a455-24fd8e17d0d3-audit-log\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.990354 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.990329    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/801b8517-ff16-41b5-a455-24fd8e17d0d3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.990548 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.990531    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/801b8517-ff16-41b5-a455-24fd8e17d0d3-metrics-server-audit-profiles\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.991964 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.991946    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/801b8517-ff16-41b5-a455-24fd8e17d0d3-client-ca-bundle\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.992372 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.992348    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/801b8517-ff16-41b5-a455-24fd8e17d0d3-secret-metrics-server-tls\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.992481 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.992434    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/801b8517-ff16-41b5-a455-24fd8e17d0d3-secret-metrics-server-client-certs\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:41.999748 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:41.999724    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnwhd\" (UniqueName: \"kubernetes.io/projected/801b8517-ff16-41b5-a455-24fd8e17d0d3-kube-api-access-cnwhd\") pod \"metrics-server-67b697bddd-26n9n\" (UID: \"801b8517-ff16-41b5-a455-24fd8e17d0d3\") " pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:42.038167 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.038130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:42.140317 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.140274    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:16:42.281230 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.281198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:42.295851 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.295712    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8pslv" podStartSLOduration=4.227218144 podStartE2EDuration="5.295691039s" totalImagesPullingTime="1.003631702s" totalInitContainerRuntime="64.841193ms" isStatefulPod=true podCreationTimestamp="2026-07-09 11:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:16:42.020926555 +0000 UTC m=+285.612657719" watchObservedRunningTime="2026-07-09 11:16:42.295691039 +0000 UTC m=+285.887422204"
Jul 09 11:16:42.296174 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.296154    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67b697bddd-26n9n"]
Jul 09 11:16:42.299153 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:16:42.299131    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801b8517_ff16_41b5_a455_24fd8e17d0d3.slice/crio-2ee03e101c72d2ec4f744108e3ee3bcaf48c8f913ddbf4ff17316c1ca7b06c14 WatchSource:0}: Error finding container 2ee03e101c72d2ec4f744108e3ee3bcaf48c8f913ddbf4ff17316c1ca7b06c14: Status 404 returned error can't find the container with id 2ee03e101c72d2ec4f744108e3ee3bcaf48c8f913ddbf4ff17316c1ca7b06c14
Jul 09 11:16:42.498810 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.498753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:42.690747 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.690661    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6447f946f9-vgk4k"]
Jul 09 11:16:42.695048 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.695030    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.698345 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.698322    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 09 11:16:42.698470 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.698423    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 09 11:16:42.698877 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.698857    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lfpf5\""
Jul 09 11:16:42.743674 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.743648    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 09 11:16:42.743945 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.743927    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 09 11:16:42.744029 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.743964    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 09 11:16:42.775692 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.775662    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6447f946f9-vgk4k"]
Jul 09 11:16:42.795500 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.795474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzv7d\" (UniqueName: \"kubernetes.io/projected/bd31e459-5084-4036-be6b-8690ad6879f0-kube-api-access-bzv7d\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.795601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.795504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bd31e459-5084-4036-be6b-8690ad6879f0-federate-client-tls\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.795601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.795534    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd31e459-5084-4036-be6b-8690ad6879f0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.795601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.795577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd31e459-5084-4036-be6b-8690ad6879f0-metrics-client-ca\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.795687 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.795629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bd31e459-5084-4036-be6b-8690ad6879f0-secret-telemeter-client\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.795687 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.795662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd31e459-5084-4036-be6b-8690ad6879f0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.795768 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.795687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bd31e459-5084-4036-be6b-8690ad6879f0-telemeter-client-tls\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.795768 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.795734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd31e459-5084-4036-be6b-8690ad6879f0-serving-certs-ca-bundle\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.806668 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.806649    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 09 11:16:42.896922 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.896887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd31e459-5084-4036-be6b-8690ad6879f0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.896922 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.896922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd31e459-5084-4036-be6b-8690ad6879f0-metrics-client-ca\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.897125 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.896957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bd31e459-5084-4036-be6b-8690ad6879f0-secret-telemeter-client\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.897125 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.896991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd31e459-5084-4036-be6b-8690ad6879f0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.897125 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.897013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bd31e459-5084-4036-be6b-8690ad6879f0-telemeter-client-tls\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.897125 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.897028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd31e459-5084-4036-be6b-8690ad6879f0-serving-certs-ca-bundle\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.897125 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.897084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzv7d\" (UniqueName: \"kubernetes.io/projected/bd31e459-5084-4036-be6b-8690ad6879f0-kube-api-access-bzv7d\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.897125 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.897122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bd31e459-5084-4036-be6b-8690ad6879f0-federate-client-tls\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.897858 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.897825    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd31e459-5084-4036-be6b-8690ad6879f0-metrics-client-ca\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.897858 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.897849    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd31e459-5084-4036-be6b-8690ad6879f0-serving-certs-ca-bundle\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.898087 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.898065    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd31e459-5084-4036-be6b-8690ad6879f0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.899527 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.899503    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd31e459-5084-4036-be6b-8690ad6879f0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.899621 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.899588    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bd31e459-5084-4036-be6b-8690ad6879f0-telemeter-client-tls\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.899754 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.899734    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bd31e459-5084-4036-be6b-8690ad6879f0-federate-client-tls\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.899815 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.899757    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bd31e459-5084-4036-be6b-8690ad6879f0-secret-telemeter-client\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.911039 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.911017    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzv7d\" (UniqueName: \"kubernetes.io/projected/bd31e459-5084-4036-be6b-8690ad6879f0-kube-api-access-bzv7d\") pod \"telemeter-client-6447f946f9-vgk4k\" (UID: \"bd31e459-5084-4036-be6b-8690ad6879f0\") " pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:42.993899 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:42.993815    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67b697bddd-26n9n" event={"ID":"801b8517-ff16-41b5-a455-24fd8e17d0d3","Type":"ContainerStarted","Data":"2ee03e101c72d2ec4f744108e3ee3bcaf48c8f913ddbf4ff17316c1ca7b06c14"}
Jul 09 11:16:43.004534 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:43.004506    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k"
Jul 09 11:16:43.182572 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:43.182530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6447f946f9-vgk4k"]
Jul 09 11:16:43.187634 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:16:43.187600    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd31e459_5084_4036_be6b_8690ad6879f0.slice/crio-98ba221704536cca558694547dc2859da700fcec3f0f2eccc8e9c74863dc047a WatchSource:0}: Error finding container 98ba221704536cca558694547dc2859da700fcec3f0f2eccc8e9c74863dc047a: Status 404 returned error can't find the container with id 98ba221704536cca558694547dc2859da700fcec3f0f2eccc8e9c74863dc047a
Jul 09 11:16:43.450801 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:43.450736    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:43.639518 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:43.639475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:43.794664 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:43.794629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:43.953816 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:43.953763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:43.998842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:43.998811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k" event={"ID":"bd31e459-5084-4036-be6b-8690ad6879f0","Type":"ContainerStarted","Data":"98ba221704536cca558694547dc2859da700fcec3f0f2eccc8e9c74863dc047a"}
Jul 09 11:16:44.157962 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:44.157883    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:44.268676 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:44.268637    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:44.409387 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:44.409293    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:44.616324 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:44.616289    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:44.787995 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:44.787959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:44.906405 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:44.906368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:45.004766 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:45.004726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67b697bddd-26n9n" event={"ID":"801b8517-ff16-41b5-a455-24fd8e17d0d3","Type":"ContainerStarted","Data":"0163e4813c74de2d8b35715b5d680a526301380befe98183ebbefc074e90412f"}
Jul 09 11:16:45.037723 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:45.037671    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-67b697bddd-26n9n" podStartSLOduration=2.401350714 podStartE2EDuration="4.037653685s" totalImagesPullingTime="1.636302971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:16:45.035142092 +0000 UTC m=+288.626873256" watchObservedRunningTime="2026-07-09 11:16:45.037653685 +0000 UTC m=+288.629384849"
Jul 09 11:16:45.169377 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:45.169302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:45.302592 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:45.302558    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:45.789828 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:45.789769    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:45.974639 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:45.974559    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:46.011409 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:46.011379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k" event={"ID":"bd31e459-5084-4036-be6b-8690ad6879f0","Type":"ContainerStarted","Data":"07483cde8c88143e42da46cc29bc3be3dfe16b1d78406a2dda94bf08a9f2882e"}
Jul 09 11:16:46.078376 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:46.078338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:46.453501 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:46.453464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:46.583446 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:46.583418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:47.016049 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:47.016021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k" event={"ID":"bd31e459-5084-4036-be6b-8690ad6879f0","Type":"ContainerStarted","Data":"d7b80af45da2331ebed48643075615b8c5a9038cd57c70e695f245f9b138f36d"}
Jul 09 11:16:47.016518 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:47.016081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k" event={"ID":"bd31e459-5084-4036-be6b-8690ad6879f0","Type":"ContainerStarted","Data":"4c950bd1b8bf4583a335b0f67c4d7845f996bba4f579a1289399024d1efc31e1"}
Jul 09 11:16:47.049925 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:47.049869    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6447f946f9-vgk4k" podStartSLOduration=1.706542138 podStartE2EDuration="5.049849105s" totalImagesPullingTime="3.343306967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:16:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:16:47.048203811 +0000 UTC m=+290.639934988" watchObservedRunningTime="2026-07-09 11:16:47.049849105 +0000 UTC m=+290.641580271"
Jul 09 11:16:48.981979 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:48.981948    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fmjjv"
Jul 09 11:16:50.445043 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:50.445006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:50.627049 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:50.627015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:50.793156 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:50.793128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:51.269859 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:51.269822    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx" podUID="64f6df1d-0636-4395-8c10-023950c23414" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 11:16:52.650158 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:52.650122    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:52.794394 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:52.794351    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:55.111237 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:55.111191    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:55.270222 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:55.270161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:56.438161 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:56.438126    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:56.611769 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:56.611736    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:56.784924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:56.784896    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:16:56.785216 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:56.785200    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:16:56.790503 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:56.790482    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 11:16:58.450658 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:58.450614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:58.653164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:58.653123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:58.842122 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:58.842076    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:16:58.942951 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:16:58.942903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:01.270163 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:01.270125    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx" podUID="64f6df1d-0636-4395-8c10-023950c23414" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 11:17:01.612197 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:01.612108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:01.752555 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:01.752498    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:02.141359 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:02.141320    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:17:02.141574 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:02.141407    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:17:03.239394 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:03.239365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:03.356555 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:03.356523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:04.964601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:04.964567    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:05.040207 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:05.040180    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:11.270139 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:11.270095    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx" podUID="64f6df1d-0636-4395-8c10-023950c23414" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 11:17:11.270508 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:11.270167    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx"
Jul 09 11:17:11.270757 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:11.270726    2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"60439454af8291ee916c608aa65b397b92fdd9b0371b31e7a9e25fd454c7b2eb"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 09 11:17:11.270813 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:11.270798    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx" podUID="64f6df1d-0636-4395-8c10-023950c23414" containerName="service-proxy" containerID="cri-o://60439454af8291ee916c608aa65b397b92fdd9b0371b31e7a9e25fd454c7b2eb" gracePeriod=30
Jul 09 11:17:12.102308 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:12.102276    2563 generic.go:356] "Generic (PLEG): container finished" podID="64f6df1d-0636-4395-8c10-023950c23414" containerID="60439454af8291ee916c608aa65b397b92fdd9b0371b31e7a9e25fd454c7b2eb" exitCode=2
Jul 09 11:17:12.102508 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:12.102350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx" event={"ID":"64f6df1d-0636-4395-8c10-023950c23414","Type":"ContainerDied","Data":"60439454af8291ee916c608aa65b397b92fdd9b0371b31e7a9e25fd454c7b2eb"}
Jul 09 11:17:12.102508 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:12.102402    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b54648947-c24rx" event={"ID":"64f6df1d-0636-4395-8c10-023950c23414","Type":"ContainerStarted","Data":"b5d8a074c96027475e112387853257e9e171bf98a52f3798360d844dd58f137e"}
Jul 09 11:17:13.221947 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:13.221917    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:16.805662 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:16.805629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:16.902464 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:16.902431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:19.960685 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:19.960653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:20.064711 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:20.064679    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:22.145905 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:22.145876    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:17:22.149884 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:22.149862    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-67b697bddd-26n9n"
Jul 09 11:17:28.332201 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:28.332166    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:28.448344 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:28.448313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:28.554175 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:28.554145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:34.977052 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:34.977016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:35.103985 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:35.103954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:51.089846 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:51.089738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:51.164215 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:51.164182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:51.698220 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:51.698182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:51.797389 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:51.797355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:53.557285 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:53.557201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:53.725347 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:53.725313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:58.350510 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:58.350471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:17:58.451205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:17:58.451167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:18:03.903803 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:18:03.903750    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:18:04.003751 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:18:04.003716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:18:04.954219 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:18:04.954191    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:18:05.092715 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:18:05.092675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:18:15.257019 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:18:15.256983    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:18:15.357641 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:18:15.357609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:18:15.429084 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:18:15.429055    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:18:25.791354 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:18:25.791320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:18:30.231931 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:18:30.231895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:18:30.290455 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:18:30.290422    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:00.578161 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:00.578123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:00.737160 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:00.737131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:27.735255 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:27.735225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:29.162103 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:29.162064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:31.338393 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:31.338359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:31.441461 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:31.441430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:32.898155 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:32.898120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:33.029356 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:33.029322    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:33.595896 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:33.595864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:33.695411 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:33.695381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:33.974952 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:33.974874    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"]
Jul 09 11:19:33.977240 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:33.977223    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"
Jul 09 11:19:33.981045 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:33.981021    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rpj9v\""
Jul 09 11:19:33.981166 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:33.981129    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 09 11:19:33.982076 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:33.982057    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 09 11:19:33.998757 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:33.998729    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"]
Jul 09 11:19:34.039159 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:34.039123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb274a86-bdae-4651-b905-a12217cc1199-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl\" (UID: \"bb274a86-bdae-4651-b905-a12217cc1199\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"
Jul 09 11:19:34.039159 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:34.039158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb274a86-bdae-4651-b905-a12217cc1199-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl\" (UID: \"bb274a86-bdae-4651-b905-a12217cc1199\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"
Jul 09 11:19:34.039383 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:34.039179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwkfc\" (UniqueName: \"kubernetes.io/projected/bb274a86-bdae-4651-b905-a12217cc1199-kube-api-access-fwkfc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl\" (UID: \"bb274a86-bdae-4651-b905-a12217cc1199\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"
Jul 09 11:19:34.139616 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:34.139586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb274a86-bdae-4651-b905-a12217cc1199-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl\" (UID: \"bb274a86-bdae-4651-b905-a12217cc1199\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"
Jul 09 11:19:34.139616 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:34.139617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb274a86-bdae-4651-b905-a12217cc1199-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl\" (UID: \"bb274a86-bdae-4651-b905-a12217cc1199\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"
Jul 09 11:19:34.139861 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:34.139638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwkfc\" (UniqueName: \"kubernetes.io/projected/bb274a86-bdae-4651-b905-a12217cc1199-kube-api-access-fwkfc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl\" (UID: \"bb274a86-bdae-4651-b905-a12217cc1199\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"
Jul 09 11:19:34.140027 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:34.140005    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb274a86-bdae-4651-b905-a12217cc1199-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl\" (UID: \"bb274a86-bdae-4651-b905-a12217cc1199\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"
Jul 09 11:19:34.140098 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:34.140053    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb274a86-bdae-4651-b905-a12217cc1199-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl\" (UID: \"bb274a86-bdae-4651-b905-a12217cc1199\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"
Jul 09 11:19:34.148084 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:34.148063    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwkfc\" (UniqueName: \"kubernetes.io/projected/bb274a86-bdae-4651-b905-a12217cc1199-kube-api-access-fwkfc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl\" (UID: \"bb274a86-bdae-4651-b905-a12217cc1199\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"
Jul 09 11:19:34.286694 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:34.286659    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"
Jul 09 11:19:34.430972 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:34.430948    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"]
Jul 09 11:19:34.433494 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:19:34.433460    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb274a86_bdae_4651_b905_a12217cc1199.slice/crio-4fbcec016d5cff2faf3987b3112780742e2c6be0d0ccab66113e375636c4807e WatchSource:0}: Error finding container 4fbcec016d5cff2faf3987b3112780742e2c6be0d0ccab66113e375636c4807e: Status 404 returned error can't find the container with id 4fbcec016d5cff2faf3987b3112780742e2c6be0d0ccab66113e375636c4807e
Jul 09 11:19:34.435436 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:34.435423    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 11:19:34.567826 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:34.567726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl" event={"ID":"bb274a86-bdae-4651-b905-a12217cc1199","Type":"ContainerStarted","Data":"4fbcec016d5cff2faf3987b3112780742e2c6be0d0ccab66113e375636c4807e"}
Jul 09 11:19:35.595273 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:35.595239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:35.859640 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:35.859534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:38.162311 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:38.162278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:38.240314 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:38.240277    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:41.595614 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:41.595579    2563 generic.go:356] "Generic (PLEG): container finished" podID="bb274a86-bdae-4651-b905-a12217cc1199" containerID="1cfc3bba344ede75b0c81bc315864dc2abbe7746d3b18b80ecc7fbbf1fcc91ab" exitCode=0
Jul 09 11:19:41.596081 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:41.595661    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl" event={"ID":"bb274a86-bdae-4651-b905-a12217cc1199","Type":"ContainerDied","Data":"1cfc3bba344ede75b0c81bc315864dc2abbe7746d3b18b80ecc7fbbf1fcc91ab"}
Jul 09 11:19:43.188909 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:43.188874    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:43.604905 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:43.604877    2563 generic.go:356] "Generic (PLEG): container finished" podID="bb274a86-bdae-4651-b905-a12217cc1199" containerID="864b622f079e829c9c4eed74624d4feeeb081b512c4d85168623f956a9196798" exitCode=0
Jul 09 11:19:43.605028 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:43.604955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl" event={"ID":"bb274a86-bdae-4651-b905-a12217cc1199","Type":"ContainerDied","Data":"864b622f079e829c9c4eed74624d4feeeb081b512c4d85168623f956a9196798"}
Jul 09 11:19:44.132322 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:44.132288    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-p449x"]
Jul 09 11:19:44.135271 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:44.135254    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-p449x"
Jul 09 11:19:44.137699 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:44.137676    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 09 11:19:44.137824 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:44.137729    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5nww5\""
Jul 09 11:19:44.138807 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:44.138738    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 09 11:19:44.143775 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:44.143753    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-p449x"]
Jul 09 11:19:44.216465 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:44.216419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a396ff2-0f8f-4d9e-a4a1-caa04508d58d-bound-sa-token\") pod \"cert-manager-55b4d954d5-p449x\" (UID: \"1a396ff2-0f8f-4d9e-a4a1-caa04508d58d\") " pod="cert-manager/cert-manager-55b4d954d5-p449x"
Jul 09 11:19:44.216925 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:44.216546    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqcv2\" (UniqueName: \"kubernetes.io/projected/1a396ff2-0f8f-4d9e-a4a1-caa04508d58d-kube-api-access-dqcv2\") pod \"cert-manager-55b4d954d5-p449x\" (UID: \"1a396ff2-0f8f-4d9e-a4a1-caa04508d58d\") " pod="cert-manager/cert-manager-55b4d954d5-p449x"
Jul 09 11:19:44.317629 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:44.317595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqcv2\" (UniqueName: \"kubernetes.io/projected/1a396ff2-0f8f-4d9e-a4a1-caa04508d58d-kube-api-access-dqcv2\") pod \"cert-manager-55b4d954d5-p449x\" (UID: \"1a396ff2-0f8f-4d9e-a4a1-caa04508d58d\") " pod="cert-manager/cert-manager-55b4d954d5-p449x"
Jul 09 11:19:44.317852 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:44.317646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a396ff2-0f8f-4d9e-a4a1-caa04508d58d-bound-sa-token\") pod \"cert-manager-55b4d954d5-p449x\" (UID: \"1a396ff2-0f8f-4d9e-a4a1-caa04508d58d\") " pod="cert-manager/cert-manager-55b4d954d5-p449x"
Jul 09 11:19:44.326137 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:44.326091    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a396ff2-0f8f-4d9e-a4a1-caa04508d58d-bound-sa-token\") pod \"cert-manager-55b4d954d5-p449x\" (UID: \"1a396ff2-0f8f-4d9e-a4a1-caa04508d58d\") " pod="cert-manager/cert-manager-55b4d954d5-p449x"
Jul 09 11:19:44.326284 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:44.326216    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqcv2\" (UniqueName: \"kubernetes.io/projected/1a396ff2-0f8f-4d9e-a4a1-caa04508d58d-kube-api-access-dqcv2\") pod \"cert-manager-55b4d954d5-p449x\" (UID: \"1a396ff2-0f8f-4d9e-a4a1-caa04508d58d\") " pod="cert-manager/cert-manager-55b4d954d5-p449x"
Jul 09 11:19:44.445300 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:44.445211    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-p449x"
Jul 09 11:19:44.591244 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:44.591208    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-p449x"]
Jul 09 11:19:44.595086 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:19:44.595055    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a396ff2_0f8f_4d9e_a4a1_caa04508d58d.slice/crio-1fdfc31285611dc9023743d05f17f64e6cdd3c186bcf8d023974daff203f0194 WatchSource:0}: Error finding container 1fdfc31285611dc9023743d05f17f64e6cdd3c186bcf8d023974daff203f0194: Status 404 returned error can't find the container with id 1fdfc31285611dc9023743d05f17f64e6cdd3c186bcf8d023974daff203f0194
Jul 09 11:19:44.610176 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:44.610140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-p449x" event={"ID":"1a396ff2-0f8f-4d9e-a4a1-caa04508d58d","Type":"ContainerStarted","Data":"1fdfc31285611dc9023743d05f17f64e6cdd3c186bcf8d023974daff203f0194"}
Jul 09 11:19:48.626982 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:48.626943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-p449x" event={"ID":"1a396ff2-0f8f-4d9e-a4a1-caa04508d58d","Type":"ContainerStarted","Data":"5f4e66d4b8c001a0a9370ce563c9cb34b85d163c5d19b5acacc75bfeabe0a4c9"}
Jul 09 11:19:48.644309 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:48.644250    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-p449x" podStartSLOduration=1.607503664 podStartE2EDuration="4.644231116s" totalImagesPullingTime="3.036727452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:19:48.643610407 +0000 UTC m=+472.235341572" watchObservedRunningTime="2026-07-09 11:19:48.644231116 +0000 UTC m=+472.235962280"
Jul 09 11:19:51.641521 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:51.641435    2563 generic.go:356] "Generic (PLEG): container finished" podID="bb274a86-bdae-4651-b905-a12217cc1199" containerID="1e2affe2e04b38271835275d3a3f74aea1dc3d31b247149d9a8801f6dfeb202c" exitCode=0
Jul 09 11:19:51.641900 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:51.641518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl" event={"ID":"bb274a86-bdae-4651-b905-a12217cc1199","Type":"ContainerDied","Data":"1e2affe2e04b38271835275d3a3f74aea1dc3d31b247149d9a8801f6dfeb202c"}
Jul 09 11:19:52.767576 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:52.767552    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"
Jul 09 11:19:52.886924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:52.886888    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb274a86-bdae-4651-b905-a12217cc1199-bundle\") pod \"bb274a86-bdae-4651-b905-a12217cc1199\" (UID: \"bb274a86-bdae-4651-b905-a12217cc1199\") "
Jul 09 11:19:52.886924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:52.886925    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb274a86-bdae-4651-b905-a12217cc1199-util\") pod \"bb274a86-bdae-4651-b905-a12217cc1199\" (UID: \"bb274a86-bdae-4651-b905-a12217cc1199\") "
Jul 09 11:19:52.887126 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:52.886970    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwkfc\" (UniqueName: \"kubernetes.io/projected/bb274a86-bdae-4651-b905-a12217cc1199-kube-api-access-fwkfc\") pod \"bb274a86-bdae-4651-b905-a12217cc1199\" (UID: \"bb274a86-bdae-4651-b905-a12217cc1199\") "
Jul 09 11:19:52.887311 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:52.887289    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb274a86-bdae-4651-b905-a12217cc1199-bundle" (OuterVolumeSpecName: "bundle") pod "bb274a86-bdae-4651-b905-a12217cc1199" (UID: "bb274a86-bdae-4651-b905-a12217cc1199"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:19:52.889271 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:52.889241    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb274a86-bdae-4651-b905-a12217cc1199-kube-api-access-fwkfc" (OuterVolumeSpecName: "kube-api-access-fwkfc") pod "bb274a86-bdae-4651-b905-a12217cc1199" (UID: "bb274a86-bdae-4651-b905-a12217cc1199"). InnerVolumeSpecName "kube-api-access-fwkfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:19:52.890898 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:52.890876    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb274a86-bdae-4651-b905-a12217cc1199-util" (OuterVolumeSpecName: "util") pod "bb274a86-bdae-4651-b905-a12217cc1199" (UID: "bb274a86-bdae-4651-b905-a12217cc1199"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:19:52.988263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:52.988225    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb274a86-bdae-4651-b905-a12217cc1199-bundle\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:19:52.988263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:52.988255    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb274a86-bdae-4651-b905-a12217cc1199-util\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:19:52.988263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:52.988265    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwkfc\" (UniqueName: \"kubernetes.io/projected/bb274a86-bdae-4651-b905-a12217cc1199-kube-api-access-fwkfc\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:19:53.651755 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:53.651724    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl"
Jul 09 11:19:53.651941 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:53.651728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg7ccl" event={"ID":"bb274a86-bdae-4651-b905-a12217cc1199","Type":"ContainerDied","Data":"4fbcec016d5cff2faf3987b3112780742e2c6be0d0ccab66113e375636c4807e"}
Jul 09 11:19:53.651941 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:53.651844    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fbcec016d5cff2faf3987b3112780742e2c6be0d0ccab66113e375636c4807e"
Jul 09 11:19:57.305925 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:57.305891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:57.388738 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:57.388709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:19:59.387240 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:19:59.387193    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:03.733429 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:03.733399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:03.871697 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:03.871669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:05.480414 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.480379    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"]
Jul 09 11:20:05.480802 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.480617    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb274a86-bdae-4651-b905-a12217cc1199" containerName="pull"
Jul 09 11:20:05.480802 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.480628    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb274a86-bdae-4651-b905-a12217cc1199" containerName="pull"
Jul 09 11:20:05.480802 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.480636    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb274a86-bdae-4651-b905-a12217cc1199" containerName="extract"
Jul 09 11:20:05.480802 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.480642    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb274a86-bdae-4651-b905-a12217cc1199" containerName="extract"
Jul 09 11:20:05.480802 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.480660    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb274a86-bdae-4651-b905-a12217cc1199" containerName="util"
Jul 09 11:20:05.480802 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.480665    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb274a86-bdae-4651-b905-a12217cc1199" containerName="util"
Jul 09 11:20:05.480802 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.480705    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb274a86-bdae-4651-b905-a12217cc1199" containerName="extract"
Jul 09 11:20:05.532632 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.532586    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"]
Jul 09 11:20:05.532814 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.532664    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"
Jul 09 11:20:05.535294 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.535273    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 09 11:20:05.535442 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.535299    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 09 11:20:05.536289 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.536274    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rpj9v\""
Jul 09 11:20:05.583105 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.583062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/245560bf-efa9-4f32-af63-7adf5ccdc113-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp\" (UID: \"245560bf-efa9-4f32-af63-7adf5ccdc113\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"
Jul 09 11:20:05.583105 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.583107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/245560bf-efa9-4f32-af63-7adf5ccdc113-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp\" (UID: \"245560bf-efa9-4f32-af63-7adf5ccdc113\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"
Jul 09 11:20:05.583334 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.583145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6hxr\" (UniqueName: \"kubernetes.io/projected/245560bf-efa9-4f32-af63-7adf5ccdc113-kube-api-access-c6hxr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp\" (UID: \"245560bf-efa9-4f32-af63-7adf5ccdc113\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"
Jul 09 11:20:05.684159 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.684119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6hxr\" (UniqueName: \"kubernetes.io/projected/245560bf-efa9-4f32-af63-7adf5ccdc113-kube-api-access-c6hxr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp\" (UID: \"245560bf-efa9-4f32-af63-7adf5ccdc113\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"
Jul 09 11:20:05.684337 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.684294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/245560bf-efa9-4f32-af63-7adf5ccdc113-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp\" (UID: \"245560bf-efa9-4f32-af63-7adf5ccdc113\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"
Jul 09 11:20:05.684395 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.684342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/245560bf-efa9-4f32-af63-7adf5ccdc113-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp\" (UID: \"245560bf-efa9-4f32-af63-7adf5ccdc113\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"
Jul 09 11:20:05.684699 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.684680    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/245560bf-efa9-4f32-af63-7adf5ccdc113-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp\" (UID: \"245560bf-efa9-4f32-af63-7adf5ccdc113\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"
Jul 09 11:20:05.684751 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.684731    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/245560bf-efa9-4f32-af63-7adf5ccdc113-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp\" (UID: \"245560bf-efa9-4f32-af63-7adf5ccdc113\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"
Jul 09 11:20:05.693189 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.693165    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6hxr\" (UniqueName: \"kubernetes.io/projected/245560bf-efa9-4f32-af63-7adf5ccdc113-kube-api-access-c6hxr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp\" (UID: \"245560bf-efa9-4f32-af63-7adf5ccdc113\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"
Jul 09 11:20:05.842256 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.842213    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"
Jul 09 11:20:05.986122 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:05.986097    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"]
Jul 09 11:20:05.988635 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:20:05.988604    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245560bf_efa9_4f32_af63_7adf5ccdc113.slice/crio-4fe0555cfe219d5608428deac254c017eb0271f716ffb329f15f9f390b635bb7 WatchSource:0}: Error finding container 4fe0555cfe219d5608428deac254c017eb0271f716ffb329f15f9f390b635bb7: Status 404 returned error can't find the container with id 4fe0555cfe219d5608428deac254c017eb0271f716ffb329f15f9f390b635bb7
Jul 09 11:20:06.253471 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:06.253437    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:06.532286 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:06.532252    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:06.607184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:06.607143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:06.697344 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:06.697321    2563 generic.go:356] "Generic (PLEG): container finished" podID="245560bf-efa9-4f32-af63-7adf5ccdc113" containerID="cab86c5d5b29289fbdd705e8c61c96e711283088e36955f56d066d365c2cdc60" exitCode=0
Jul 09 11:20:06.697423 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:06.697401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp" event={"ID":"245560bf-efa9-4f32-af63-7adf5ccdc113","Type":"ContainerDied","Data":"cab86c5d5b29289fbdd705e8c61c96e711283088e36955f56d066d365c2cdc60"}
Jul 09 11:20:06.697472 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:06.697460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp" event={"ID":"245560bf-efa9-4f32-af63-7adf5ccdc113","Type":"ContainerStarted","Data":"4fe0555cfe219d5608428deac254c017eb0271f716ffb329f15f9f390b635bb7"}
Jul 09 11:20:08.709683 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:08.709650    2563 generic.go:356] "Generic (PLEG): container finished" podID="245560bf-efa9-4f32-af63-7adf5ccdc113" containerID="add716f34ee9757eda8b11c00d00b4fd01cc1addba560df24d239f2ebc047c7b" exitCode=0
Jul 09 11:20:08.710186 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:08.709717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp" event={"ID":"245560bf-efa9-4f32-af63-7adf5ccdc113","Type":"ContainerDied","Data":"add716f34ee9757eda8b11c00d00b4fd01cc1addba560df24d239f2ebc047c7b"}
Jul 09 11:20:09.714770 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:09.714739    2563 generic.go:356] "Generic (PLEG): container finished" podID="245560bf-efa9-4f32-af63-7adf5ccdc113" containerID="90316d8684c2e32b888cd5e3fac728b96e56b6cb6c9d092fe02d843e069a6e7c" exitCode=0
Jul 09 11:20:09.715153 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:09.714776    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp" event={"ID":"245560bf-efa9-4f32-af63-7adf5ccdc113","Type":"ContainerDied","Data":"90316d8684c2e32b888cd5e3fac728b96e56b6cb6c9d092fe02d843e069a6e7c"}
Jul 09 11:20:10.844430 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:10.844407    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"
Jul 09 11:20:10.925642 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:10.925604    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/245560bf-efa9-4f32-af63-7adf5ccdc113-util\") pod \"245560bf-efa9-4f32-af63-7adf5ccdc113\" (UID: \"245560bf-efa9-4f32-af63-7adf5ccdc113\") "
Jul 09 11:20:10.925840 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:10.925660    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/245560bf-efa9-4f32-af63-7adf5ccdc113-bundle\") pod \"245560bf-efa9-4f32-af63-7adf5ccdc113\" (UID: \"245560bf-efa9-4f32-af63-7adf5ccdc113\") "
Jul 09 11:20:10.925840 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:10.925686    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6hxr\" (UniqueName: \"kubernetes.io/projected/245560bf-efa9-4f32-af63-7adf5ccdc113-kube-api-access-c6hxr\") pod \"245560bf-efa9-4f32-af63-7adf5ccdc113\" (UID: \"245560bf-efa9-4f32-af63-7adf5ccdc113\") "
Jul 09 11:20:10.926389 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:10.926352    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245560bf-efa9-4f32-af63-7adf5ccdc113-bundle" (OuterVolumeSpecName: "bundle") pod "245560bf-efa9-4f32-af63-7adf5ccdc113" (UID: "245560bf-efa9-4f32-af63-7adf5ccdc113"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:20:10.927799 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:10.927764    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245560bf-efa9-4f32-af63-7adf5ccdc113-kube-api-access-c6hxr" (OuterVolumeSpecName: "kube-api-access-c6hxr") pod "245560bf-efa9-4f32-af63-7adf5ccdc113" (UID: "245560bf-efa9-4f32-af63-7adf5ccdc113"). InnerVolumeSpecName "kube-api-access-c6hxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:20:10.931560 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:10.931527    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245560bf-efa9-4f32-af63-7adf5ccdc113-util" (OuterVolumeSpecName: "util") pod "245560bf-efa9-4f32-af63-7adf5ccdc113" (UID: "245560bf-efa9-4f32-af63-7adf5ccdc113"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:20:11.026844 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:11.026811    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/245560bf-efa9-4f32-af63-7adf5ccdc113-util\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:20:11.026844 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:11.026844    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/245560bf-efa9-4f32-af63-7adf5ccdc113-bundle\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:20:11.027027 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:11.026855    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6hxr\" (UniqueName: \"kubernetes.io/projected/245560bf-efa9-4f32-af63-7adf5ccdc113-kube-api-access-c6hxr\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:20:11.725581 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:11.725550    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp" event={"ID":"245560bf-efa9-4f32-af63-7adf5ccdc113","Type":"ContainerDied","Data":"4fe0555cfe219d5608428deac254c017eb0271f716ffb329f15f9f390b635bb7"}
Jul 09 11:20:11.725738 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:11.725594    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afffndp"
Jul 09 11:20:11.725738 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:11.725600    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fe0555cfe219d5608428deac254c017eb0271f716ffb329f15f9f390b635bb7"
Jul 09 11:20:12.518752 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.518719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:12.638042 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.638010    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:12.683244 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.683214    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"]
Jul 09 11:20:12.683452 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.683441    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245560bf-efa9-4f32-af63-7adf5ccdc113" containerName="extract"
Jul 09 11:20:12.683496 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.683454    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="245560bf-efa9-4f32-af63-7adf5ccdc113" containerName="extract"
Jul 09 11:20:12.683496 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.683474    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245560bf-efa9-4f32-af63-7adf5ccdc113" containerName="pull"
Jul 09 11:20:12.683496 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.683480    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="245560bf-efa9-4f32-af63-7adf5ccdc113" containerName="pull"
Jul 09 11:20:12.683496 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.683488    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245560bf-efa9-4f32-af63-7adf5ccdc113" containerName="util"
Jul 09 11:20:12.683496 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.683493    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="245560bf-efa9-4f32-af63-7adf5ccdc113" containerName="util"
Jul 09 11:20:12.683639 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.683532    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="245560bf-efa9-4f32-af63-7adf5ccdc113" containerName="extract"
Jul 09 11:20:12.685312 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.685296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:12.688542 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.688519    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-khblb\""
Jul 09 11:20:12.688542 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.688528    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 09 11:20:12.689674 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.689653    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 09 11:20:12.689899 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.689886    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 09 11:20:12.690032 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.690018    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 09 11:20:12.690168 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.690154    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 09 11:20:12.698429 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.698408    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"]
Jul 09 11:20:12.739419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.739387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj2f5\" (UniqueName: \"kubernetes.io/projected/e71125e1-b9e6-42b6-855c-f256a120169c-kube-api-access-pj2f5\") pod \"lws-controller-manager-cbdc75774-xf7gg\" (UID: \"e71125e1-b9e6-42b6-855c-f256a120169c\") " pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:12.739619 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.739441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e71125e1-b9e6-42b6-855c-f256a120169c-metrics-cert\") pod \"lws-controller-manager-cbdc75774-xf7gg\" (UID: \"e71125e1-b9e6-42b6-855c-f256a120169c\") " pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:12.739619 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.739541    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e71125e1-b9e6-42b6-855c-f256a120169c-manager-config\") pod \"lws-controller-manager-cbdc75774-xf7gg\" (UID: \"e71125e1-b9e6-42b6-855c-f256a120169c\") " pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:12.739619 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.739581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e71125e1-b9e6-42b6-855c-f256a120169c-cert\") pod \"lws-controller-manager-cbdc75774-xf7gg\" (UID: \"e71125e1-b9e6-42b6-855c-f256a120169c\") " pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:12.841050 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.840941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e71125e1-b9e6-42b6-855c-f256a120169c-manager-config\") pod \"lws-controller-manager-cbdc75774-xf7gg\" (UID: \"e71125e1-b9e6-42b6-855c-f256a120169c\") " pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:12.841050 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.840999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e71125e1-b9e6-42b6-855c-f256a120169c-cert\") pod \"lws-controller-manager-cbdc75774-xf7gg\" (UID: \"e71125e1-b9e6-42b6-855c-f256a120169c\") " pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:12.841050 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.841035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj2f5\" (UniqueName: \"kubernetes.io/projected/e71125e1-b9e6-42b6-855c-f256a120169c-kube-api-access-pj2f5\") pod \"lws-controller-manager-cbdc75774-xf7gg\" (UID: \"e71125e1-b9e6-42b6-855c-f256a120169c\") " pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:12.841322 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.841111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e71125e1-b9e6-42b6-855c-f256a120169c-metrics-cert\") pod \"lws-controller-manager-cbdc75774-xf7gg\" (UID: \"e71125e1-b9e6-42b6-855c-f256a120169c\") " pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:12.841584 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.841563    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e71125e1-b9e6-42b6-855c-f256a120169c-manager-config\") pod \"lws-controller-manager-cbdc75774-xf7gg\" (UID: \"e71125e1-b9e6-42b6-855c-f256a120169c\") " pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:12.843620 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.843592    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e71125e1-b9e6-42b6-855c-f256a120169c-cert\") pod \"lws-controller-manager-cbdc75774-xf7gg\" (UID: \"e71125e1-b9e6-42b6-855c-f256a120169c\") " pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:12.843727 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.843684    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e71125e1-b9e6-42b6-855c-f256a120169c-metrics-cert\") pod \"lws-controller-manager-cbdc75774-xf7gg\" (UID: \"e71125e1-b9e6-42b6-855c-f256a120169c\") " pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:12.849867 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.849844    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj2f5\" (UniqueName: \"kubernetes.io/projected/e71125e1-b9e6-42b6-855c-f256a120169c-kube-api-access-pj2f5\") pod \"lws-controller-manager-cbdc75774-xf7gg\" (UID: \"e71125e1-b9e6-42b6-855c-f256a120169c\") " pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:12.994222 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:12.994188    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:13.118501 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:13.118428    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"]
Jul 09 11:20:13.121472 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:20:13.121440    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71125e1_b9e6_42b6_855c_f256a120169c.slice/crio-419a571e3c7c32f7dee22617ce86f08cd688abc35e686e8eca61598ace0c4b5d WatchSource:0}: Error finding container 419a571e3c7c32f7dee22617ce86f08cd688abc35e686e8eca61598ace0c4b5d: Status 404 returned error can't find the container with id 419a571e3c7c32f7dee22617ce86f08cd688abc35e686e8eca61598ace0c4b5d
Jul 09 11:20:13.733438 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:13.733397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg" event={"ID":"e71125e1-b9e6-42b6-855c-f256a120169c","Type":"ContainerStarted","Data":"419a571e3c7c32f7dee22617ce86f08cd688abc35e686e8eca61598ace0c4b5d"}
Jul 09 11:20:15.748044 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:15.748007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg" event={"ID":"e71125e1-b9e6-42b6-855c-f256a120169c","Type":"ContainerStarted","Data":"64c9559dc9072fa11a8162da62f4a5d20b477a90f463bda93e56a24632a9a063"}
Jul 09 11:20:15.748548 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:15.748150    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:15.766412 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:15.766360    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg" podStartSLOduration=2.021941349 podStartE2EDuration="3.766345088s" totalImagesPullingTime="1.744403739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:20:15.764804554 +0000 UTC m=+499.356535717" watchObservedRunningTime="2026-07-09 11:20:15.766345088 +0000 UTC m=+499.358076251"
Jul 09 11:20:17.794212 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:17.794180    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:17.899738 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:17.899707    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:18.416146 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.416115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:18.615232 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.615197    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:18.720404 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.720321    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:18.840502 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.840463    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"]
Jul 09 11:20:18.843066 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.843041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:18.850602 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.850570    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 09 11:20:18.850860 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.850843    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 09 11:20:18.851072 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.851057    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 09 11:20:18.851412 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.851394    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5vbbz\""
Jul 09 11:20:18.851985 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.851962    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 09 11:20:18.918749 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.918703    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"]
Jul 09 11:20:18.990205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.990112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/92da1819-00ee-4b3f-b105-25428d8fda49-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65b4d55d4c-9vnpk\" (UID: \"92da1819-00ee-4b3f-b105-25428d8fda49\") " pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:18.990205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.990168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/92da1819-00ee-4b3f-b105-25428d8fda49-manifests\") pod \"opendatahub-operator-controller-manager-65b4d55d4c-9vnpk\" (UID: \"92da1819-00ee-4b3f-b105-25428d8fda49\") " pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:18.990205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.990185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92da1819-00ee-4b3f-b105-25428d8fda49-tmp\") pod \"opendatahub-operator-controller-manager-65b4d55d4c-9vnpk\" (UID: \"92da1819-00ee-4b3f-b105-25428d8fda49\") " pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:18.990205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.990203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w5pq\" (UniqueName: \"kubernetes.io/projected/92da1819-00ee-4b3f-b105-25428d8fda49-kube-api-access-4w5pq\") pod \"opendatahub-operator-controller-manager-65b4d55d4c-9vnpk\" (UID: \"92da1819-00ee-4b3f-b105-25428d8fda49\") " pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:18.990500 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:18.990267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92da1819-00ee-4b3f-b105-25428d8fda49-webhook-cert\") pod \"opendatahub-operator-controller-manager-65b4d55d4c-9vnpk\" (UID: \"92da1819-00ee-4b3f-b105-25428d8fda49\") " pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:19.023980 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.023945    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:19.091202 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.091170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/92da1819-00ee-4b3f-b105-25428d8fda49-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65b4d55d4c-9vnpk\" (UID: \"92da1819-00ee-4b3f-b105-25428d8fda49\") " pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:19.091336 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.091225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/92da1819-00ee-4b3f-b105-25428d8fda49-manifests\") pod \"opendatahub-operator-controller-manager-65b4d55d4c-9vnpk\" (UID: \"92da1819-00ee-4b3f-b105-25428d8fda49\") " pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:19.091336 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.091244    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92da1819-00ee-4b3f-b105-25428d8fda49-tmp\") pod \"opendatahub-operator-controller-manager-65b4d55d4c-9vnpk\" (UID: \"92da1819-00ee-4b3f-b105-25428d8fda49\") " pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:19.091336 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.091263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w5pq\" (UniqueName: \"kubernetes.io/projected/92da1819-00ee-4b3f-b105-25428d8fda49-kube-api-access-4w5pq\") pod \"opendatahub-operator-controller-manager-65b4d55d4c-9vnpk\" (UID: \"92da1819-00ee-4b3f-b105-25428d8fda49\") " pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:19.091336 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.091309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92da1819-00ee-4b3f-b105-25428d8fda49-webhook-cert\") pod \"opendatahub-operator-controller-manager-65b4d55d4c-9vnpk\" (UID: \"92da1819-00ee-4b3f-b105-25428d8fda49\") " pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:19.091701 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.091644    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/92da1819-00ee-4b3f-b105-25428d8fda49-manifests\") pod \"opendatahub-operator-controller-manager-65b4d55d4c-9vnpk\" (UID: \"92da1819-00ee-4b3f-b105-25428d8fda49\") " pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:19.093724 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.093674    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92da1819-00ee-4b3f-b105-25428d8fda49-tmp\") pod \"opendatahub-operator-controller-manager-65b4d55d4c-9vnpk\" (UID: \"92da1819-00ee-4b3f-b105-25428d8fda49\") " pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:19.093862 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.093753    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/92da1819-00ee-4b3f-b105-25428d8fda49-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65b4d55d4c-9vnpk\" (UID: \"92da1819-00ee-4b3f-b105-25428d8fda49\") " pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:19.093921 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.093896    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92da1819-00ee-4b3f-b105-25428d8fda49-webhook-cert\") pod \"opendatahub-operator-controller-manager-65b4d55d4c-9vnpk\" (UID: \"92da1819-00ee-4b3f-b105-25428d8fda49\") " pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:19.099228 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.099208    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w5pq\" (UniqueName: \"kubernetes.io/projected/92da1819-00ee-4b3f-b105-25428d8fda49-kube-api-access-4w5pq\") pod \"opendatahub-operator-controller-manager-65b4d55d4c-9vnpk\" (UID: \"92da1819-00ee-4b3f-b105-25428d8fda49\") " pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:19.124693 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.124667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:19.172671 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.172638    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:19.225176 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.225140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:19.310597 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.310565    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"]
Jul 09 11:20:19.313302 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.313278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:19.313945 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:20:19.313920    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92da1819_00ee_4b3f_b105_25428d8fda49.slice/crio-ba1887c0bc678d2d55a81549d7d4aac8d2dcd819a0d5f32604f8adbf94eb041b WatchSource:0}: Error finding container ba1887c0bc678d2d55a81549d7d4aac8d2dcd819a0d5f32604f8adbf94eb041b: Status 404 returned error can't find the container with id ba1887c0bc678d2d55a81549d7d4aac8d2dcd819a0d5f32604f8adbf94eb041b
Jul 09 11:20:19.406390 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.406359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:19.497350 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.497319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:19.764636 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:19.764596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk" event={"ID":"92da1819-00ee-4b3f-b105-25428d8fda49","Type":"ContainerStarted","Data":"ba1887c0bc678d2d55a81549d7d4aac8d2dcd819a0d5f32604f8adbf94eb041b"}
Jul 09 11:20:20.547642 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:20.547606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:22.777999 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:22.777966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk" event={"ID":"92da1819-00ee-4b3f-b105-25428d8fda49","Type":"ContainerStarted","Data":"630dff43f9c2fd8b6c05bc8b20cefe51dbc5e0c46971ea217297dd5c74e5eff8"}
Jul 09 11:20:23.784034 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:23.783995    2563 generic.go:356] "Generic (PLEG): container finished" podID="92da1819-00ee-4b3f-b105-25428d8fda49" containerID="630dff43f9c2fd8b6c05bc8b20cefe51dbc5e0c46971ea217297dd5c74e5eff8" exitCode=0
Jul 09 11:20:23.784409 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:23.784082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk" event={"ID":"92da1819-00ee-4b3f-b105-25428d8fda49","Type":"ContainerDied","Data":"630dff43f9c2fd8b6c05bc8b20cefe51dbc5e0c46971ea217297dd5c74e5eff8"}
Jul 09 11:20:24.789903 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:24.789860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk" event={"ID":"92da1819-00ee-4b3f-b105-25428d8fda49","Type":"ContainerStarted","Data":"c1d48178792ef50ae49765896813ad2258f72b55c982154fbf611a4b8211f95e"}
Jul 09 11:20:24.790362 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:24.789972    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:24.814935 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:24.814882    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk" podStartSLOduration=2.404507514 podStartE2EDuration="6.814866143s" totalImagesPullingTime="3.399651941s" totalInitContainerRuntime="1.010706688s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:20:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:20:24.813006565 +0000 UTC m=+508.404737743" watchObservedRunningTime="2026-07-09 11:20:24.814866143 +0000 UTC m=+508.406597372"
Jul 09 11:20:26.754175 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:26.754148    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-cbdc75774-xf7gg"
Jul 09 11:20:35.795996 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:35.795961    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-65b4d55d4c-9vnpk"
Jul 09 11:20:37.728630 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:37.728596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:37.848735 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:37.848698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:38.609217 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:38.609184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:38.726257 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:38.726219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:38.826838 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:38.826809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:38.936044 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:38.935936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:39.382478 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:39.382451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:39.558899 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:39.558866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:43.550873 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:43.550838    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:43.651993 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:43.651964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:44.534085 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:44.534049    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:44.637681 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:44.637648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:44.974806 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:44.974711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:45.050581 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:45.050546    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:46.442844 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:46.442747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:46.732466 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:46.732374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:46.833358 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:46.833320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:48.016536 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.016504    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"]
Jul 09 11:20:48.022292 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.022272    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.024999 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.024975    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 09 11:20:48.025105 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.024976    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 09 11:20:48.025283 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.025268    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 09 11:20:48.025737 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.025719    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 09 11:20:48.025833 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.025801    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vgn56\""
Jul 09 11:20:48.025993 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.025955    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 11:20:48.026495 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.026483    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 09 11:20:48.036340 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.036308    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"]
Jul 09 11:20:48.117436 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.117405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7582b03-1757-439f-ae07-2112955e472e-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.117621 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.117456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7582b03-1757-439f-ae07-2112955e472e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.117621 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.117479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7582b03-1757-439f-ae07-2112955e472e-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.117621 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.117535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7582b03-1757-439f-ae07-2112955e472e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.117621 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.117565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26jc\" (UniqueName: \"kubernetes.io/projected/a7582b03-1757-439f-ae07-2112955e472e-kube-api-access-g26jc\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.117621 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.117594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7582b03-1757-439f-ae07-2112955e472e-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.117621 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.117612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7582b03-1757-439f-ae07-2112955e472e-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.218026 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.217989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7582b03-1757-439f-ae07-2112955e472e-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.218151 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.218036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7582b03-1757-439f-ae07-2112955e472e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.218151 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.218054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26jc\" (UniqueName: \"kubernetes.io/projected/a7582b03-1757-439f-ae07-2112955e472e-kube-api-access-g26jc\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.218151 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.218101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7582b03-1757-439f-ae07-2112955e472e-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.218151 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.218128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7582b03-1757-439f-ae07-2112955e472e-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.218364 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.218160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7582b03-1757-439f-ae07-2112955e472e-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.218364 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.218227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7582b03-1757-439f-ae07-2112955e472e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.218850 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.218829    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a7582b03-1757-439f-ae07-2112955e472e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.220547 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.220519    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7582b03-1757-439f-ae07-2112955e472e-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.220659 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.220570    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a7582b03-1757-439f-ae07-2112955e472e-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.220743 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.220724    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7582b03-1757-439f-ae07-2112955e472e-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.220967 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.220947    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a7582b03-1757-439f-ae07-2112955e472e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.227720 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.227694    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26jc\" (UniqueName: \"kubernetes.io/projected/a7582b03-1757-439f-ae07-2112955e472e-kube-api-access-g26jc\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.232279 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.232258    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7582b03-1757-439f-ae07-2112955e472e-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-hm9wl\" (UID: \"a7582b03-1757-439f-ae07-2112955e472e\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.331257 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.331147    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:48.476274 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.476244    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"]
Jul 09 11:20:48.479390 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:20:48.479351    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7582b03_1757_439f_ae07_2112955e472e.slice/crio-6cdb1c05a722ec5c8f83255b87d4c6b1fdc91bc2d501447856efa3adbfc62f5a WatchSource:0}: Error finding container 6cdb1c05a722ec5c8f83255b87d4c6b1fdc91bc2d501447856efa3adbfc62f5a: Status 404 returned error can't find the container with id 6cdb1c05a722ec5c8f83255b87d4c6b1fdc91bc2d501447856efa3adbfc62f5a
Jul 09 11:20:48.887526 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:48.887481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl" event={"ID":"a7582b03-1757-439f-ae07-2112955e472e","Type":"ContainerStarted","Data":"6cdb1c05a722ec5c8f83255b87d4c6b1fdc91bc2d501447856efa3adbfc62f5a"}
Jul 09 11:20:49.090397 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:49.090365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:49.224403 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:49.224328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:49.627368 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:49.627296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:49.727338 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:49.727307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:50.438359 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:50.438320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:50.592635 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:50.592605    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:50.637435 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:50.637394    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:20:50.637539 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:50.637481    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:20:50.688263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:50.688240    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:50.898329 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:50.898297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl" event={"ID":"a7582b03-1757-439f-ae07-2112955e472e","Type":"ContainerStarted","Data":"7e2d3ee67ac8c3b789adff1ec24e10c4bc99f79a7199986540797f1f1f2c5bbb"}
Jul 09 11:20:50.898594 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:50.898576    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:20:50.924207 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:50.924153    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl" podStartSLOduration=1.768219894 podStartE2EDuration="3.924136706s" totalImagesPullingTime="2.155916812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:20:50.922582161 +0000 UTC m=+534.514313351" watchObservedRunningTime="2026-07-09 11:20:50.924136706 +0000 UTC m=+534.515867870"
Jul 09 11:20:51.312708 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:51.312671    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:51.451092 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:51.451059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:51.550050 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:51.550020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:51.650299 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:51.650210    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:20:51.904622 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:20:51.904545    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hm9wl"
Jul 09 11:21:39.263310 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:21:39.263275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:21:41.195254 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:21:41.195224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:21:52.619738 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:21:52.619704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:21:56.812743 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:21:56.812717    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:21:56.814144 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:21:56.814122    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:22:06.784125 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:06.784090    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:10.258214 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:10.258181    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:11.783885 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:11.783847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:12.186736 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:12.186655    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:13.265973 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:13.265942    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:13.408122 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:13.408089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:14.796983 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:14.796944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:15.854068 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:15.854035    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:17.125158 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:17.125086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:17.279310 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:17.279271    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:17.377487 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:17.377404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:17.749220 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:17.749139    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:17.854903 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:17.854870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:18.546398 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:18.546369    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:21.098754 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:21.098721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:21.192675 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:21.192641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:21.606847 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:21.606814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:22.176113 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:22.176080    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:27.659949 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:27.659907    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:27.761934 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:27.761887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:27.863754 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:27.863721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:31.076651 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:31.076617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:31.208607 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:31.208562    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:31.335677 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:31.335587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:34.016908 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:34.016870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:34.145093 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:34.145065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:36.525514 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:36.525471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:36.640739 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:36.640709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:42.233911 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:42.233882    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:42.372189 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:42.372160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:50.870836 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:50.870804    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:50.970878 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:50.970843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:53.883568 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:53.883529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:53.986211 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:53.986179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:56.373275 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:56.373242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:22:56.474962 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:22:56.474930    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:02.944166 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:02.944130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:03.293385 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:03.293353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:04.751457 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.751423    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:04.754982 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.754952    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fbt27"]
Jul 09 11:23:04.756900 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.756883    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27"
Jul 09 11:23:04.759461 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.759438    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-t7dgp\""
Jul 09 11:23:04.759562 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.759438    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 09 11:23:04.759562 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.759440    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 11:23:04.760373 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.760356    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 11:23:04.768833 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.768816    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fbt27"]
Jul 09 11:23:04.800556 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.800530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7fa8826f-1721-4818-8471-5ad31e3b7838-config-file\") pod \"limitador-limitador-64c94fd64f-fbt27\" (UID: \"7fa8826f-1721-4818-8471-5ad31e3b7838\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27"
Jul 09 11:23:04.800660 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.800561    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9glk\" (UniqueName: \"kubernetes.io/projected/7fa8826f-1721-4818-8471-5ad31e3b7838-kube-api-access-n9glk\") pod \"limitador-limitador-64c94fd64f-fbt27\" (UID: \"7fa8826f-1721-4818-8471-5ad31e3b7838\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27"
Jul 09 11:23:04.850721 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.850687    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fbt27"]
Jul 09 11:23:04.876276 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.876248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:04.901548 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.901523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7fa8826f-1721-4818-8471-5ad31e3b7838-config-file\") pod \"limitador-limitador-64c94fd64f-fbt27\" (UID: \"7fa8826f-1721-4818-8471-5ad31e3b7838\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27"
Jul 09 11:23:04.901657 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.901553    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9glk\" (UniqueName: \"kubernetes.io/projected/7fa8826f-1721-4818-8471-5ad31e3b7838-kube-api-access-n9glk\") pod \"limitador-limitador-64c94fd64f-fbt27\" (UID: \"7fa8826f-1721-4818-8471-5ad31e3b7838\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27"
Jul 09 11:23:04.902203 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.902184    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7fa8826f-1721-4818-8471-5ad31e3b7838-config-file\") pod \"limitador-limitador-64c94fd64f-fbt27\" (UID: \"7fa8826f-1721-4818-8471-5ad31e3b7838\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27"
Jul 09 11:23:04.909320 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.909301    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9glk\" (UniqueName: \"kubernetes.io/projected/7fa8826f-1721-4818-8471-5ad31e3b7838-kube-api-access-n9glk\") pod \"limitador-limitador-64c94fd64f-fbt27\" (UID: \"7fa8826f-1721-4818-8471-5ad31e3b7838\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27"
Jul 09 11:23:04.937302 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:04.937276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:05.067314 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:05.067230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27"
Jul 09 11:23:05.193329 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:05.193302    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fbt27"]
Jul 09 11:23:05.194981 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:23:05.194954    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa8826f_1721_4818_8471_5ad31e3b7838.slice/crio-67e4a7164e087776c33e84698d102c71d48327bad403f017c233e19d695c6a1e WatchSource:0}: Error finding container 67e4a7164e087776c33e84698d102c71d48327bad403f017c233e19d695c6a1e: Status 404 returned error can't find the container with id 67e4a7164e087776c33e84698d102c71d48327bad403f017c233e19d695c6a1e
Jul 09 11:23:05.437255 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:05.437168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27" event={"ID":"7fa8826f-1721-4818-8471-5ad31e3b7838","Type":"ContainerStarted","Data":"67e4a7164e087776c33e84698d102c71d48327bad403f017c233e19d695c6a1e"}
Jul 09 11:23:05.589947 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:05.589922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:05.667172 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:05.667140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:08.452213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:08.452178    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27" event={"ID":"7fa8826f-1721-4818-8471-5ad31e3b7838","Type":"ContainerStarted","Data":"1b277d345117abaaed39fbc5fb7a561cda99a26afe66bb0c22c531c12f5213f2"}
Jul 09 11:23:08.452625 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:08.452309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27"
Jul 09 11:23:08.473507 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:08.473450    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27" podStartSLOduration=1.838612483 podStartE2EDuration="4.473435699s" totalImagesPullingTime="2.634823216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:23:08.471179991 +0000 UTC m=+672.062911156" watchObservedRunningTime="2026-07-09 11:23:08.473435699 +0000 UTC m=+672.065166863"
Jul 09 11:23:19.457602 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:19.457562    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27"
Jul 09 11:23:21.789416 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:21.789376    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fbt27"]
Jul 09 11:23:21.789852 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:21.789569    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27" podUID="7fa8826f-1721-4818-8471-5ad31e3b7838" containerName="limitador" containerID="cri-o://1b277d345117abaaed39fbc5fb7a561cda99a26afe66bb0c22c531c12f5213f2" gracePeriod=30
Jul 09 11:23:22.509311 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:22.509280    2563 generic.go:356] "Generic (PLEG): container finished" podID="7fa8826f-1721-4818-8471-5ad31e3b7838" containerID="1b277d345117abaaed39fbc5fb7a561cda99a26afe66bb0c22c531c12f5213f2" exitCode=0
Jul 09 11:23:22.509484 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:22.509352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27" event={"ID":"7fa8826f-1721-4818-8471-5ad31e3b7838","Type":"ContainerDied","Data":"1b277d345117abaaed39fbc5fb7a561cda99a26afe66bb0c22c531c12f5213f2"}
Jul 09 11:23:22.731012 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:22.730986    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27"
Jul 09 11:23:22.852045 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:22.851939    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7fa8826f-1721-4818-8471-5ad31e3b7838-config-file\") pod \"7fa8826f-1721-4818-8471-5ad31e3b7838\" (UID: \"7fa8826f-1721-4818-8471-5ad31e3b7838\") "
Jul 09 11:23:22.852045 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:22.852000    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9glk\" (UniqueName: \"kubernetes.io/projected/7fa8826f-1721-4818-8471-5ad31e3b7838-kube-api-access-n9glk\") pod \"7fa8826f-1721-4818-8471-5ad31e3b7838\" (UID: \"7fa8826f-1721-4818-8471-5ad31e3b7838\") "
Jul 09 11:23:22.852554 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:22.852297    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa8826f-1721-4818-8471-5ad31e3b7838-config-file" (OuterVolumeSpecName: "config-file") pod "7fa8826f-1721-4818-8471-5ad31e3b7838" (UID: "7fa8826f-1721-4818-8471-5ad31e3b7838"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:23:22.854326 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:22.854301    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa8826f-1721-4818-8471-5ad31e3b7838-kube-api-access-n9glk" (OuterVolumeSpecName: "kube-api-access-n9glk") pod "7fa8826f-1721-4818-8471-5ad31e3b7838" (UID: "7fa8826f-1721-4818-8471-5ad31e3b7838"). InnerVolumeSpecName "kube-api-access-n9glk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:23:22.952754 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:22.952714    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7fa8826f-1721-4818-8471-5ad31e3b7838-config-file\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:23:22.952754 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:22.952739    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9glk\" (UniqueName: \"kubernetes.io/projected/7fa8826f-1721-4818-8471-5ad31e3b7838-kube-api-access-n9glk\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:23:23.514235 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:23.514201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27" event={"ID":"7fa8826f-1721-4818-8471-5ad31e3b7838","Type":"ContainerDied","Data":"67e4a7164e087776c33e84698d102c71d48327bad403f017c233e19d695c6a1e"}
Jul 09 11:23:23.514418 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:23.514261    2563 scope.go:117] "RemoveContainer" containerID="1b277d345117abaaed39fbc5fb7a561cda99a26afe66bb0c22c531c12f5213f2"
Jul 09 11:23:23.514418 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:23.514207    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fbt27"
Jul 09 11:23:23.531426 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:23.531404    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fbt27"]
Jul 09 11:23:23.533655 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:23.533634    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fbt27"]
Jul 09 11:23:24.949716 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:24.949682    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fa8826f-1721-4818-8471-5ad31e3b7838" path="/var/lib/kubelet/pods/7fa8826f-1721-4818-8471-5ad31e3b7838/volumes"
Jul 09 11:23:26.006326 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.006296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:26.102468 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.102437    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:26.429243 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.429163    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5"]
Jul 09 11:23:26.429490 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.429475    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa8826f-1721-4818-8471-5ad31e3b7838" containerName="limitador"
Jul 09 11:23:26.429539 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.429494    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa8826f-1721-4818-8471-5ad31e3b7838" containerName="limitador"
Jul 09 11:23:26.429576 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.429553    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa8826f-1721-4818-8471-5ad31e3b7838" containerName="limitador"
Jul 09 11:23:26.432259 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.432243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5"
Jul 09 11:23:26.435869 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.435847    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 11:23:26.436149 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.436135    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-2vkks\""
Jul 09 11:23:26.436202 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.436175    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 09 11:23:26.436423 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.436409    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 11:23:26.441274 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.441254    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5"]
Jul 09 11:23:26.464310 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.464281    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:26.482491 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.482462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e8ed-57f9-4a77-a215-c414c05186b7-data\") pod \"postgres-69fb4594cf-kdjm5\" (UID: \"f5a7e8ed-57f9-4a77-a215-c414c05186b7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5"
Jul 09 11:23:26.482584 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.482502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbmwj\" (UniqueName: \"kubernetes.io/projected/f5a7e8ed-57f9-4a77-a215-c414c05186b7-kube-api-access-hbmwj\") pod \"postgres-69fb4594cf-kdjm5\" (UID: \"f5a7e8ed-57f9-4a77-a215-c414c05186b7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5"
Jul 09 11:23:26.583695 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.583657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbmwj\" (UniqueName: \"kubernetes.io/projected/f5a7e8ed-57f9-4a77-a215-c414c05186b7-kube-api-access-hbmwj\") pod \"postgres-69fb4594cf-kdjm5\" (UID: \"f5a7e8ed-57f9-4a77-a215-c414c05186b7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5"
Jul 09 11:23:26.583884 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.583763    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e8ed-57f9-4a77-a215-c414c05186b7-data\") pod \"postgres-69fb4594cf-kdjm5\" (UID: \"f5a7e8ed-57f9-4a77-a215-c414c05186b7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5"
Jul 09 11:23:26.584136 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.584119    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f5a7e8ed-57f9-4a77-a215-c414c05186b7-data\") pod \"postgres-69fb4594cf-kdjm5\" (UID: \"f5a7e8ed-57f9-4a77-a215-c414c05186b7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5"
Jul 09 11:23:26.591967 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.591945    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbmwj\" (UniqueName: \"kubernetes.io/projected/f5a7e8ed-57f9-4a77-a215-c414c05186b7-kube-api-access-hbmwj\") pod \"postgres-69fb4594cf-kdjm5\" (UID: \"f5a7e8ed-57f9-4a77-a215-c414c05186b7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5"
Jul 09 11:23:26.744054 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.743965    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5"
Jul 09 11:23:26.875929 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:26.875904    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5"]
Jul 09 11:23:26.877431 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:23:26.877408    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a7e8ed_57f9_4a77_a215_c414c05186b7.slice/crio-1830637d8b715e84c30c24735c91bae54161d02decc2d2ba28bd55795596db71 WatchSource:0}: Error finding container 1830637d8b715e84c30c24735c91bae54161d02decc2d2ba28bd55795596db71: Status 404 returned error can't find the container with id 1830637d8b715e84c30c24735c91bae54161d02decc2d2ba28bd55795596db71
Jul 09 11:23:27.539436 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:27.539404    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5" event={"ID":"f5a7e8ed-57f9-4a77-a215-c414c05186b7","Type":"ContainerStarted","Data":"1830637d8b715e84c30c24735c91bae54161d02decc2d2ba28bd55795596db71"}
Jul 09 11:23:32.566130 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:32.566093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5" event={"ID":"f5a7e8ed-57f9-4a77-a215-c414c05186b7","Type":"ContainerStarted","Data":"31a7fbfdf4e84f06c57c159293d032126732b4c1bcc0c93ccbc26245fd04f894"}
Jul 09 11:23:32.566495 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:32.566216    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5"
Jul 09 11:23:32.585059 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:32.584967    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5" podStartSLOduration=1.8943713500000001 podStartE2EDuration="6.584947354s" totalImagesPullingTime="4.690576004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:23:32.584047169 +0000 UTC m=+696.175778334" watchObservedRunningTime="2026-07-09 11:23:32.584947354 +0000 UTC m=+696.176678518"
Jul 09 11:23:38.598819 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:38.598763    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kdjm5"
Jul 09 11:23:38.968796 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:38.968706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:39.196100 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:39.196073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:45.972414 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:45.972376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:45.977650 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:45.977622    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-96f784fd6-bwltg"]
Jul 09 11:23:45.997535 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:45.997498    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-96f784fd6-bwltg"
Jul 09 11:23:46.000423 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.000401    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-96f784fd6-bwltg"]
Jul 09 11:23:46.000700 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.000671    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 09 11:23:46.000886 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.000862    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 09 11:23:46.001459 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.001440    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d9swf\""
Jul 09 11:23:46.063390 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.063355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:46.156044 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.156010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52baa97d-1544-4353-9885-c1670fd14988-webhook-cert\") pod \"maas-controller-96f784fd6-bwltg\" (UID: \"52baa97d-1544-4353-9885-c1670fd14988\") " pod="opendatahub/maas-controller-96f784fd6-bwltg"
Jul 09 11:23:46.156209 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.156059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx6x2\" (UniqueName: \"kubernetes.io/projected/52baa97d-1544-4353-9885-c1670fd14988-kube-api-access-gx6x2\") pod \"maas-controller-96f784fd6-bwltg\" (UID: \"52baa97d-1544-4353-9885-c1670fd14988\") " pod="opendatahub/maas-controller-96f784fd6-bwltg"
Jul 09 11:23:46.188321 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.188275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:46.256930 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.256900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx6x2\" (UniqueName: \"kubernetes.io/projected/52baa97d-1544-4353-9885-c1670fd14988-kube-api-access-gx6x2\") pod \"maas-controller-96f784fd6-bwltg\" (UID: \"52baa97d-1544-4353-9885-c1670fd14988\") " pod="opendatahub/maas-controller-96f784fd6-bwltg"
Jul 09 11:23:46.257084 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.257008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52baa97d-1544-4353-9885-c1670fd14988-webhook-cert\") pod \"maas-controller-96f784fd6-bwltg\" (UID: \"52baa97d-1544-4353-9885-c1670fd14988\") " pod="opendatahub/maas-controller-96f784fd6-bwltg"
Jul 09 11:23:46.259325 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.259304    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52baa97d-1544-4353-9885-c1670fd14988-webhook-cert\") pod \"maas-controller-96f784fd6-bwltg\" (UID: \"52baa97d-1544-4353-9885-c1670fd14988\") " pod="opendatahub/maas-controller-96f784fd6-bwltg"
Jul 09 11:23:46.268004 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.267979    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx6x2\" (UniqueName: \"kubernetes.io/projected/52baa97d-1544-4353-9885-c1670fd14988-kube-api-access-gx6x2\") pod \"maas-controller-96f784fd6-bwltg\" (UID: \"52baa97d-1544-4353-9885-c1670fd14988\") " pod="opendatahub/maas-controller-96f784fd6-bwltg"
Jul 09 11:23:46.276433 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.276403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:46.310589 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.310559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-96f784fd6-bwltg"
Jul 09 11:23:46.445065 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.445039    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-96f784fd6-bwltg"]
Jul 09 11:23:46.447443 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:23:46.447418    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52baa97d_1544_4353_9885_c1670fd14988.slice/crio-71e85c96f8d28c67ba6a4198943ee7439ff0894adcae17e0e6e4085040fbbaef WatchSource:0}: Error finding container 71e85c96f8d28c67ba6a4198943ee7439ff0894adcae17e0e6e4085040fbbaef: Status 404 returned error can't find the container with id 71e85c96f8d28c67ba6a4198943ee7439ff0894adcae17e0e6e4085040fbbaef
Jul 09 11:23:46.629086 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:46.628997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-96f784fd6-bwltg" event={"ID":"52baa97d-1544-4353-9885-c1670fd14988","Type":"ContainerStarted","Data":"71e85c96f8d28c67ba6a4198943ee7439ff0894adcae17e0e6e4085040fbbaef"}
Jul 09 11:23:48.851730 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:48.851692    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:48.947197 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:48.947089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:23:49.644321 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:49.644289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-96f784fd6-bwltg" event={"ID":"52baa97d-1544-4353-9885-c1670fd14988","Type":"ContainerStarted","Data":"c0760cefee418e375205392047abc045aeba2b7ac25103a51fad4560acfb2dd6"}
Jul 09 11:23:49.644551 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:49.644445    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-96f784fd6-bwltg"
Jul 09 11:23:49.691360 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:23:49.691304    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-96f784fd6-bwltg" podStartSLOduration=2.524468938 podStartE2EDuration="4.691284947s" totalImagesPullingTime="2.166816009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:23:49.690906474 +0000 UTC m=+713.282637639" watchObservedRunningTime="2026-07-09 11:23:49.691284947 +0000 UTC m=+713.283016113"
Jul 09 11:24:00.653915 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:00.653885    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-96f784fd6-bwltg"
Jul 09 11:24:01.113016 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.112980    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-96f784fd6-bwltg"]
Jul 09 11:24:01.113188 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.113164    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-96f784fd6-bwltg" podUID="52baa97d-1544-4353-9885-c1670fd14988" containerName="manager" containerID="cri-o://c0760cefee418e375205392047abc045aeba2b7ac25103a51fad4560acfb2dd6" gracePeriod=30
Jul 09 11:24:01.347041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.347014    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-96f784fd6-bwltg"
Jul 09 11:24:01.376506 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.376413    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx6x2\" (UniqueName: \"kubernetes.io/projected/52baa97d-1544-4353-9885-c1670fd14988-kube-api-access-gx6x2\") pod \"52baa97d-1544-4353-9885-c1670fd14988\" (UID: \"52baa97d-1544-4353-9885-c1670fd14988\") "
Jul 09 11:24:01.376506 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.376494    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52baa97d-1544-4353-9885-c1670fd14988-webhook-cert\") pod \"52baa97d-1544-4353-9885-c1670fd14988\" (UID: \"52baa97d-1544-4353-9885-c1670fd14988\") "
Jul 09 11:24:01.378563 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.378537    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52baa97d-1544-4353-9885-c1670fd14988-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "52baa97d-1544-4353-9885-c1670fd14988" (UID: "52baa97d-1544-4353-9885-c1670fd14988"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 11:24:01.378671 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.378593    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52baa97d-1544-4353-9885-c1670fd14988-kube-api-access-gx6x2" (OuterVolumeSpecName: "kube-api-access-gx6x2") pod "52baa97d-1544-4353-9885-c1670fd14988" (UID: "52baa97d-1544-4353-9885-c1670fd14988"). InnerVolumeSpecName "kube-api-access-gx6x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:24:01.477809 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.477748    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx6x2\" (UniqueName: \"kubernetes.io/projected/52baa97d-1544-4353-9885-c1670fd14988-kube-api-access-gx6x2\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:24:01.477809 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.477778    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52baa97d-1544-4353-9885-c1670fd14988-webhook-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:24:01.693958 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.693865    2563 generic.go:356] "Generic (PLEG): container finished" podID="52baa97d-1544-4353-9885-c1670fd14988" containerID="c0760cefee418e375205392047abc045aeba2b7ac25103a51fad4560acfb2dd6" exitCode=0
Jul 09 11:24:01.693958 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.693917    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-96f784fd6-bwltg"
Jul 09 11:24:01.694456 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.693960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-96f784fd6-bwltg" event={"ID":"52baa97d-1544-4353-9885-c1670fd14988","Type":"ContainerDied","Data":"c0760cefee418e375205392047abc045aeba2b7ac25103a51fad4560acfb2dd6"}
Jul 09 11:24:01.694456 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.694023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-96f784fd6-bwltg" event={"ID":"52baa97d-1544-4353-9885-c1670fd14988","Type":"ContainerDied","Data":"71e85c96f8d28c67ba6a4198943ee7439ff0894adcae17e0e6e4085040fbbaef"}
Jul 09 11:24:01.694456 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.694058    2563 scope.go:117] "RemoveContainer" containerID="c0760cefee418e375205392047abc045aeba2b7ac25103a51fad4560acfb2dd6"
Jul 09 11:24:01.704852 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.704835    2563 scope.go:117] "RemoveContainer" containerID="c0760cefee418e375205392047abc045aeba2b7ac25103a51fad4560acfb2dd6"
Jul 09 11:24:01.705118 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:24:01.705099    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0760cefee418e375205392047abc045aeba2b7ac25103a51fad4560acfb2dd6\": container with ID starting with c0760cefee418e375205392047abc045aeba2b7ac25103a51fad4560acfb2dd6 not found: ID does not exist" containerID="c0760cefee418e375205392047abc045aeba2b7ac25103a51fad4560acfb2dd6"
Jul 09 11:24:01.705169 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.705127    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0760cefee418e375205392047abc045aeba2b7ac25103a51fad4560acfb2dd6"} err="failed to get container status \"c0760cefee418e375205392047abc045aeba2b7ac25103a51fad4560acfb2dd6\": rpc error: code = NotFound desc = could not find container \"c0760cefee418e375205392047abc045aeba2b7ac25103a51fad4560acfb2dd6\": container with ID starting with c0760cefee418e375205392047abc045aeba2b7ac25103a51fad4560acfb2dd6 not found: ID does not exist"
Jul 09 11:24:01.716859 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.716832    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-96f784fd6-bwltg"]
Jul 09 11:24:01.718843 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:01.718819    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-96f784fd6-bwltg"]
Jul 09 11:24:02.949363 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:02.949329    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52baa97d-1544-4353-9885-c1670fd14988" path="/var/lib/kubelet/pods/52baa97d-1544-4353-9885-c1670fd14988/volumes"
Jul 09 11:24:08.495511 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:08.495474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:24:20.825844 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:20.825807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:24:21.316646 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.316609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:24:21.351441 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.351400    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"]
Jul 09 11:24:21.351723 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.351700    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52baa97d-1544-4353-9885-c1670fd14988" containerName="manager"
Jul 09 11:24:21.351723 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.351719    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="52baa97d-1544-4353-9885-c1670fd14988" containerName="manager"
Jul 09 11:24:21.351909 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.351799    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="52baa97d-1544-4353-9885-c1670fd14988" containerName="manager"
Jul 09 11:24:21.354533 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.354518    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"
Jul 09 11:24:21.356946 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.356927    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q69zr\""
Jul 09 11:24:21.357289 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.357271    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 09 11:24:21.366851 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.366830    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"]
Jul 09 11:24:21.437315 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.437286    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5tpr\" (UniqueName: \"kubernetes.io/projected/9c079833-413e-4547-a962-7a06267b2cdb-kube-api-access-c5tpr\") pod \"maas-api-c49d68f48-b4bwh\" (UID: \"9c079833-413e-4547-a962-7a06267b2cdb\") " pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"
Jul 09 11:24:21.437482 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.437341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c079833-413e-4547-a962-7a06267b2cdb-openshift-service-ca\") pod \"maas-api-c49d68f48-b4bwh\" (UID: \"9c079833-413e-4547-a962-7a06267b2cdb\") " pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"
Jul 09 11:24:21.437482 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.437414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c079833-413e-4547-a962-7a06267b2cdb-maas-api-tls\") pod \"maas-api-c49d68f48-b4bwh\" (UID: \"9c079833-413e-4547-a962-7a06267b2cdb\") " pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"
Jul 09 11:24:21.468087 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.468057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:24:21.538383 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.538345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c079833-413e-4547-a962-7a06267b2cdb-openshift-service-ca\") pod \"maas-api-c49d68f48-b4bwh\" (UID: \"9c079833-413e-4547-a962-7a06267b2cdb\") " pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"
Jul 09 11:24:21.538546 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.538391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c079833-413e-4547-a962-7a06267b2cdb-maas-api-tls\") pod \"maas-api-c49d68f48-b4bwh\" (UID: \"9c079833-413e-4547-a962-7a06267b2cdb\") " pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"
Jul 09 11:24:21.538546 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.538437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5tpr\" (UniqueName: \"kubernetes.io/projected/9c079833-413e-4547-a962-7a06267b2cdb-kube-api-access-c5tpr\") pod \"maas-api-c49d68f48-b4bwh\" (UID: \"9c079833-413e-4547-a962-7a06267b2cdb\") " pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"
Jul 09 11:24:21.539007 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.538988    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c079833-413e-4547-a962-7a06267b2cdb-openshift-service-ca\") pod \"maas-api-c49d68f48-b4bwh\" (UID: \"9c079833-413e-4547-a962-7a06267b2cdb\") " pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"
Jul 09 11:24:21.540854 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.540835    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c079833-413e-4547-a962-7a06267b2cdb-maas-api-tls\") pod \"maas-api-c49d68f48-b4bwh\" (UID: \"9c079833-413e-4547-a962-7a06267b2cdb\") " pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"
Jul 09 11:24:21.546995 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.546973    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5tpr\" (UniqueName: \"kubernetes.io/projected/9c079833-413e-4547-a962-7a06267b2cdb-kube-api-access-c5tpr\") pod \"maas-api-c49d68f48-b4bwh\" (UID: \"9c079833-413e-4547-a962-7a06267b2cdb\") " pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"
Jul 09 11:24:21.667436 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.667359    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"
Jul 09 11:24:21.810836 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.810762    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"]
Jul 09 11:24:21.812593 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:24:21.812565    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c079833_413e_4547_a962_7a06267b2cdb.slice/crio-d4dfa7ecb122bd40bbcfa292066576785f6b56751fb22695524ff0545adff002 WatchSource:0}: Error finding container d4dfa7ecb122bd40bbcfa292066576785f6b56751fb22695524ff0545adff002: Status 404 returned error can't find the container with id d4dfa7ecb122bd40bbcfa292066576785f6b56751fb22695524ff0545adff002
Jul 09 11:24:21.825636 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.825613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:24:21.926260 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:21.926175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:24:22.783905 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:22.783871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh" event={"ID":"9c079833-413e-4547-a962-7a06267b2cdb","Type":"ContainerStarted","Data":"d4dfa7ecb122bd40bbcfa292066576785f6b56751fb22695524ff0545adff002"}
Jul 09 11:24:23.794838 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:23.794797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh" event={"ID":"9c079833-413e-4547-a962-7a06267b2cdb","Type":"ContainerStarted","Data":"205df0cc88cb0a1c64a8f1caac7965444ea8c299be7f645fde2bfededbad9517"}
Jul 09 11:24:23.795345 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:23.794946    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"
Jul 09 11:24:23.814097 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:23.814050    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh" podStartSLOduration=1.249172608 podStartE2EDuration="2.814034644s" totalImagesPullingTime="1.564862036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:24:23.812826261 +0000 UTC m=+747.404557427" watchObservedRunningTime="2026-07-09 11:24:23.814034644 +0000 UTC m=+747.405765808"
Jul 09 11:24:29.804500 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:29.804459    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-c49d68f48-b4bwh"
Jul 09 11:24:30.844452 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:30.844419    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:24:30.941253 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:30.941216    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:24:35.880244 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:35.880205    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"]
Jul 09 11:24:35.882643 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:35.882620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:35.886636 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:35.886617    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 09 11:24:35.886812 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:35.886795    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t79pf\""
Jul 09 11:24:35.887640 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:35.887623    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 09 11:24:35.887694 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:35.887679    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 09 11:24:35.906480 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:35.906448    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"]
Jul 09 11:24:35.970132 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:35.970099    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:24:36.060258 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.060222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc76v\" (UniqueName: \"kubernetes.io/projected/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-kube-api-access-rc76v\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.060258 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.060258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.060446 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.060303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.060446 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.060342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.060446 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.060380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.060446 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.060425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.161596 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.161508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.161596 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.161565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.161596 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.161598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.161843 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.161639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc76v\" (UniqueName: \"kubernetes.io/projected/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-kube-api-access-rc76v\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.161843 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.161656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.161843 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.161683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.162010 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.161987    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.162116 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.162091    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.162170 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.162129    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.163931 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.163913    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.164326 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.164302    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.170836 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.170817    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc76v\" (UniqueName: \"kubernetes.io/projected/4ebc912d-4e28-44e9-a7a7-c399ee8419ee-kube-api-access-rc76v\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p\" (UID: \"4ebc912d-4e28-44e9-a7a7-c399ee8419ee\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.192752 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.192733    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:36.325157 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.325131    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"]
Jul 09 11:24:36.326653 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:24:36.326618    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebc912d_4e28_44e9_a7a7_c399ee8419ee.slice/crio-8dad695104fc4ed7c4e65243b5ef1ca6400b29019cae4139438628a9e9131fbf WatchSource:0}: Error finding container 8dad695104fc4ed7c4e65243b5ef1ca6400b29019cae4139438628a9e9131fbf: Status 404 returned error can't find the container with id 8dad695104fc4ed7c4e65243b5ef1ca6400b29019cae4139438628a9e9131fbf
Jul 09 11:24:36.328123 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.328103    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 11:24:36.848646 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:36.848611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p" event={"ID":"4ebc912d-4e28-44e9-a7a7-c399ee8419ee","Type":"ContainerStarted","Data":"8dad695104fc4ed7c4e65243b5ef1ca6400b29019cae4139438628a9e9131fbf"}
Jul 09 11:24:37.853600 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:37.853565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p" event={"ID":"4ebc912d-4e28-44e9-a7a7-c399ee8419ee","Type":"ContainerStarted","Data":"5888637aaf296511e949aede6785cb552e3e7d8ee38f04de9a83f5f0b3a8d42d"}
Jul 09 11:24:37.882841 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:37.882775    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p" podStartSLOduration=1.7141571089999998 podStartE2EDuration="2.882758529s" totalImagesPullingTime="1.16860142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:24:37.880434138 +0000 UTC m=+761.472165304" watchObservedRunningTime="2026-07-09 11:24:37.882758529 +0000 UTC m=+761.474489709"
Jul 09 11:24:41.974258 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:41.974220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:24:46.192894 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:46.192850    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:46.193274 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:46.192927    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:46.205450 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:46.205428    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:46.901943 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:46.901915    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p"
Jul 09 11:24:54.183791 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:24:54.183747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:25:05.129475 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.129439    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"]
Jul 09 11:25:05.132339 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.132322    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.134744 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.134718    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 09 11:25:05.145583 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.145561    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"]
Jul 09 11:25:05.175617 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.175579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:25:05.176710 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.176689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqn5v\" (UniqueName: \"kubernetes.io/projected/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-kube-api-access-mqn5v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.176771 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.176723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.176771 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.176763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.176875 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.176856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.176909 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.176900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.176945 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.176927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.277451 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.277401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.277630 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.277471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.277630 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.277515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.277630 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.277547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.277630 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.277588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.277851 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.277632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqn5v\" (UniqueName: \"kubernetes.io/projected/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-kube-api-access-mqn5v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.277943 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.277862    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.277943 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.277917    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.278046 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.278012    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.279739 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.279709    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.280119 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.280097    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.285750 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.285724    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqn5v\" (UniqueName: \"kubernetes.io/projected/29a7d16f-5bda-42cf-a856-8c8cb67cd6ec-kube-api-access-mqn5v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4\" (UID: \"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.445370 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.445284    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:05.576589 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.576559    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"]
Jul 09 11:25:05.578598 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:25:05.578567    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a7d16f_5bda_42cf_a856_8c8cb67cd6ec.slice/crio-7879c1b08ffdf23662c759dcf5ff4d0c3a8fd23287f97a634283c3b288a21ff8 WatchSource:0}: Error finding container 7879c1b08ffdf23662c759dcf5ff4d0c3a8fd23287f97a634283c3b288a21ff8: Status 404 returned error can't find the container with id 7879c1b08ffdf23662c759dcf5ff4d0c3a8fd23287f97a634283c3b288a21ff8
Jul 09 11:25:05.973434 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.973394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4" event={"ID":"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec","Type":"ContainerStarted","Data":"45511ae19f6b4fe43a72b41bdd690e42c31000ead4a54fabcc5ea7aed769acc6"}
Jul 09 11:25:05.973606 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.973452    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4" event={"ID":"29a7d16f-5bda-42cf-a856-8c8cb67cd6ec","Type":"ContainerStarted","Data":"7879c1b08ffdf23662c759dcf5ff4d0c3a8fd23287f97a634283c3b288a21ff8"}
Jul 09 11:25:05.992901 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:05.992853    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4" podStartSLOduration=0.787254862 podStartE2EDuration="992.837851ms" totalImagesPullingTime="205.582989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:25:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:25:05.99210485 +0000 UTC m=+789.583836014" watchObservedRunningTime="2026-07-09 11:25:05.992837851 +0000 UTC m=+789.584569015"
Jul 09 11:25:13.154477 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:13.154448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:25:15.446353 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:15.446306    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:15.446736 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:15.446408    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:15.458861 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:15.458834    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:16.031937 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:16.031912    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4"
Jul 09 11:25:30.186643 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:30.186612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:25:32.570670 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:25:32.570625    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:26:13.754682 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:26:13.754618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:26:13.820761 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:26:13.820710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:26:31.265230 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:26:31.265195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:26:31.365025 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:26:31.364988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:26:32.444243 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:26:32.444210    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:26:32.529698 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:26:32.529659    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:26:56.837764 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:26:56.837730    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:26:56.839541 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:26:56.839519    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:27:08.650827 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:08.650749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:27:09.915809 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:09.915739    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:27:30.129494 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:30.129452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:27:43.619628 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:43.619594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:27:44.883091 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:44.883058    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:27:45.023563 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:45.023530    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59bbb887b4-t7xfq"]
Jul 09 11:27:45.025792 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:45.025769    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59bbb887b4-t7xfq"
Jul 09 11:27:45.034245 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:45.034226    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d9swf\""
Jul 09 11:27:45.034332 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:45.034226    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 09 11:27:45.034332 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:45.034279    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 09 11:27:45.048437 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:45.048406    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59bbb887b4-t7xfq"]
Jul 09 11:27:45.128456 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:45.128414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmx2n\" (UniqueName: \"kubernetes.io/projected/dbe83611-f892-4f10-9263-0c654cb10c3c-kube-api-access-rmx2n\") pod \"maas-controller-59bbb887b4-t7xfq\" (UID: \"dbe83611-f892-4f10-9263-0c654cb10c3c\") " pod="opendatahub/maas-controller-59bbb887b4-t7xfq"
Jul 09 11:27:45.128656 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:45.128534    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dbe83611-f892-4f10-9263-0c654cb10c3c-webhook-cert\") pod \"maas-controller-59bbb887b4-t7xfq\" (UID: \"dbe83611-f892-4f10-9263-0c654cb10c3c\") " pod="opendatahub/maas-controller-59bbb887b4-t7xfq"
Jul 09 11:27:45.229956 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:45.229854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmx2n\" (UniqueName: \"kubernetes.io/projected/dbe83611-f892-4f10-9263-0c654cb10c3c-kube-api-access-rmx2n\") pod \"maas-controller-59bbb887b4-t7xfq\" (UID: \"dbe83611-f892-4f10-9263-0c654cb10c3c\") " pod="opendatahub/maas-controller-59bbb887b4-t7xfq"
Jul 09 11:27:45.229956 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:45.229957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dbe83611-f892-4f10-9263-0c654cb10c3c-webhook-cert\") pod \"maas-controller-59bbb887b4-t7xfq\" (UID: \"dbe83611-f892-4f10-9263-0c654cb10c3c\") " pod="opendatahub/maas-controller-59bbb887b4-t7xfq"
Jul 09 11:27:45.232415 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:45.232375    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dbe83611-f892-4f10-9263-0c654cb10c3c-webhook-cert\") pod \"maas-controller-59bbb887b4-t7xfq\" (UID: \"dbe83611-f892-4f10-9263-0c654cb10c3c\") " pod="opendatahub/maas-controller-59bbb887b4-t7xfq"
Jul 09 11:27:45.239109 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:45.239087    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmx2n\" (UniqueName: \"kubernetes.io/projected/dbe83611-f892-4f10-9263-0c654cb10c3c-kube-api-access-rmx2n\") pod \"maas-controller-59bbb887b4-t7xfq\" (UID: \"dbe83611-f892-4f10-9263-0c654cb10c3c\") " pod="opendatahub/maas-controller-59bbb887b4-t7xfq"
Jul 09 11:27:45.336212 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:45.336170    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59bbb887b4-t7xfq"
Jul 09 11:27:45.465234 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:45.465206    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59bbb887b4-t7xfq"]
Jul 09 11:27:45.468408 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:27:45.468372    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe83611_f892_4f10_9263_0c654cb10c3c.slice/crio-c9efb46ff752ee5de32a7fdcc887ea957550a5103d77a23ed0e897ffdc9f60fa WatchSource:0}: Error finding container c9efb46ff752ee5de32a7fdcc887ea957550a5103d77a23ed0e897ffdc9f60fa: Status 404 returned error can't find the container with id c9efb46ff752ee5de32a7fdcc887ea957550a5103d77a23ed0e897ffdc9f60fa
Jul 09 11:27:45.657375 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:45.657340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59bbb887b4-t7xfq" event={"ID":"dbe83611-f892-4f10-9263-0c654cb10c3c","Type":"ContainerStarted","Data":"c9efb46ff752ee5de32a7fdcc887ea957550a5103d77a23ed0e897ffdc9f60fa"}
Jul 09 11:27:46.663058 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:46.663023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59bbb887b4-t7xfq" event={"ID":"dbe83611-f892-4f10-9263-0c654cb10c3c","Type":"ContainerStarted","Data":"e3cfc362c69464006c766cfe906e90d25552d003cddf4fcc3fd477dc59dfc5ac"}
Jul 09 11:27:46.663511 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:46.663089    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59bbb887b4-t7xfq"
Jul 09 11:27:46.683305 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:46.683255    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-59bbb887b4-t7xfq" podStartSLOduration=1.981691042 podStartE2EDuration="2.683239108s" totalImagesPullingTime="701.548066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:27:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:27:46.681861223 +0000 UTC m=+950.273592388" watchObservedRunningTime="2026-07-09 11:27:46.683239108 +0000 UTC m=+950.274970271"
Jul 09 11:27:57.673198 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:27:57.673168    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59bbb887b4-t7xfq"
Jul 09 11:28:13.147100 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:28:13.147063    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:28:19.225858 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:28:19.225737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:28:19.921819 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:28:19.921764    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:28:24.044598 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:28:24.044540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:28:24.134287 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:28:24.134257    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:28:24.272188 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:28:24.272158    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:28:30.303090 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:28:30.303048    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:28:30.403411 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:28:30.403369    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:28:31.274150 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:28:31.274094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:28:31.553491 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:28:31.553405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:28:31.637199 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:28:31.637155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:28:31.739903 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:28:31.739867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:28:38.514296 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:28:38.514260    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:28:38.679410 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:28:38.679381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:30:15.694884 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:30:15.694851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:31:56.868811 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:31:56.868725    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:31:56.870981 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:31:56.870961    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:33:05.635886 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:33:05.635851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:33:06.837778 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:33:06.837744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:33:26.325418 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:33:26.325362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:34:46.403103 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:34:46.403008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:34:57.807629 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:34:57.807595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:36:56.893566 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:36:56.893532    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:36:56.897385 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:36:56.897365    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:38:29.528920 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:38:29.528881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:38:31.644862 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:38:31.644800    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:32.928042 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:32.927981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:33.041704 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:33.041672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:33.414425 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:33.414377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:39.712005 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:39.711936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:39.781005 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:39.780965    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:39.911017 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:39.910972    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"]
Jul 09 11:39:39.914948 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:39.914926    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:39.917416 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:39.917398    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5bca60b8-openshift-default-dockercfg-dpb7x\""
Jul 09 11:39:39.942084 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:39.942046    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"]
Jul 09 11:39:39.955436 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:39.955407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:40.036249 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.036207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56faf4bb-af96-4417-8a53-849a05d4367e-istio-podinfo\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.036249 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.036254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-workload-socket\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.036509 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.036272    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56faf4bb-af96-4417-8a53-849a05d4367e-istio-token\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.036509 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.036298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dp7z\" (UniqueName: \"kubernetes.io/projected/56faf4bb-af96-4417-8a53-849a05d4367e-kube-api-access-8dp7z\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.036509 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.036351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-istio-envoy\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.036509 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.036379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-workload-certs\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.036509 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.036491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-credential-socket\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.036705 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.036531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-istio-data\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.036705 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.036583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56faf4bb-af96-4417-8a53-849a05d4367e-istiod-ca-cert\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.071574 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.071533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:40.137608 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.137569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56faf4bb-af96-4417-8a53-849a05d4367e-istiod-ca-cert\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.137813 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.137627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56faf4bb-af96-4417-8a53-849a05d4367e-istio-podinfo\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.137813 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.137649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-workload-socket\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.137813 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.137665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56faf4bb-af96-4417-8a53-849a05d4367e-istio-token\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.137813 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.137683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dp7z\" (UniqueName: \"kubernetes.io/projected/56faf4bb-af96-4417-8a53-849a05d4367e-kube-api-access-8dp7z\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.137813 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.137702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-istio-envoy\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.137813 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.137739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-workload-certs\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.138144 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.137819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-credential-socket\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.138144 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.137843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-istio-data\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.138262 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.138156    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-workload-socket\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.138382 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.138359    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-workload-certs\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.138450 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.138407    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-credential-socket\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.138450 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.138417    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-istio-data\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.138549 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.138444    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56faf4bb-af96-4417-8a53-849a05d4367e-istiod-ca-cert\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.140165 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.140143    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-istio-envoy\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.140376 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.140360    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56faf4bb-af96-4417-8a53-849a05d4367e-istio-podinfo\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.148322 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.148301    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56faf4bb-af96-4417-8a53-849a05d4367e-istio-token\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.148653 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.148632    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dp7z\" (UniqueName: \"kubernetes.io/projected/56faf4bb-af96-4417-8a53-849a05d4367e-kube-api-access-8dp7z\") pod \"e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") " pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.225290 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.225259    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:40.378568 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.378526    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"]
Jul 09 11:39:40.381727 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:39:40.381685    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56faf4bb_af96_4417_8a53_849a05d4367e.slice/crio-f0037bc70b5b18eabd2d2e5f0f74a363fe36bf09e7e24f430ebde2364b22ab18 WatchSource:0}: Error finding container f0037bc70b5b18eabd2d2e5f0f74a363fe36bf09e7e24f430ebde2364b22ab18: Status 404 returned error can't find the container with id f0037bc70b5b18eabd2d2e5f0f74a363fe36bf09e7e24f430ebde2364b22ab18
Jul 09 11:39:40.384018 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.383996    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 11:39:40.715455 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:40.715373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw" event={"ID":"56faf4bb-af96-4417-8a53-849a05d4367e","Type":"ContainerStarted","Data":"f0037bc70b5b18eabd2d2e5f0f74a363fe36bf09e7e24f430ebde2364b22ab18"}
Jul 09 11:39:41.227686 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.227648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:41.265220 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.265179    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"]
Jul 09 11:39:41.268916 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.268887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:41.271988 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.271966    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5bca60b8-serving-cert\""
Jul 09 11:39:41.284507 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.284450    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"]
Jul 09 11:39:41.346675 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.346641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5664ec9e-5993-4231-a028-2b5b22d93682-maas-api-tls\") pod \"maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb\" (UID: \"5664ec9e-5993-4231-a028-2b5b22d93682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:41.346888 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.346752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhdnx\" (UniqueName: \"kubernetes.io/projected/5664ec9e-5993-4231-a028-2b5b22d93682-kube-api-access-nhdnx\") pod \"maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb\" (UID: \"5664ec9e-5993-4231-a028-2b5b22d93682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:41.346888 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.346844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5664ec9e-5993-4231-a028-2b5b22d93682-openshift-service-ca\") pod \"maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb\" (UID: \"5664ec9e-5993-4231-a028-2b5b22d93682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:41.397032 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.396968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:41.448723 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.448178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5664ec9e-5993-4231-a028-2b5b22d93682-maas-api-tls\") pod \"maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb\" (UID: \"5664ec9e-5993-4231-a028-2b5b22d93682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:41.448723 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.448285    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhdnx\" (UniqueName: \"kubernetes.io/projected/5664ec9e-5993-4231-a028-2b5b22d93682-kube-api-access-nhdnx\") pod \"maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb\" (UID: \"5664ec9e-5993-4231-a028-2b5b22d93682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:41.448723 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:39:41.448351    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-serving-cert: secret "maas-api-e2e-ait-5bca60b8-serving-cert" not found
Jul 09 11:39:41.448723 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.448354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5664ec9e-5993-4231-a028-2b5b22d93682-openshift-service-ca\") pod \"maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb\" (UID: \"5664ec9e-5993-4231-a028-2b5b22d93682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:41.448723 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:39:41.448440    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5664ec9e-5993-4231-a028-2b5b22d93682-maas-api-tls podName:5664ec9e-5993-4231-a028-2b5b22d93682 nodeName:}" failed. No retries permitted until 2026-07-09 11:39:41.948417092 +0000 UTC m=+1665.540148256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5664ec9e-5993-4231-a028-2b5b22d93682-maas-api-tls") pod "maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb" (UID: "5664ec9e-5993-4231-a028-2b5b22d93682") : secret "maas-api-e2e-ait-5bca60b8-serving-cert" not found
Jul 09 11:39:41.449150 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.449098    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5664ec9e-5993-4231-a028-2b5b22d93682-openshift-service-ca\") pod \"maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb\" (UID: \"5664ec9e-5993-4231-a028-2b5b22d93682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:41.466353 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.466317    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhdnx\" (UniqueName: \"kubernetes.io/projected/5664ec9e-5993-4231-a028-2b5b22d93682-kube-api-access-nhdnx\") pod \"maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb\" (UID: \"5664ec9e-5993-4231-a028-2b5b22d93682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:41.808838 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.808807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:41.912041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.912001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:41.954407 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:41.954367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5664ec9e-5993-4231-a028-2b5b22d93682-maas-api-tls\") pod \"maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb\" (UID: \"5664ec9e-5993-4231-a028-2b5b22d93682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:41.954614 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:39:41.954546    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-serving-cert: secret "maas-api-e2e-ait-5bca60b8-serving-cert" not found
Jul 09 11:39:41.954702 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:39:41.954626    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5664ec9e-5993-4231-a028-2b5b22d93682-maas-api-tls podName:5664ec9e-5993-4231-a028-2b5b22d93682 nodeName:}" failed. No retries permitted until 2026-07-09 11:39:42.954604998 +0000 UTC m=+1666.546336154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5664ec9e-5993-4231-a028-2b5b22d93682-maas-api-tls") pod "maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb" (UID: "5664ec9e-5993-4231-a028-2b5b22d93682") : secret "maas-api-e2e-ait-5bca60b8-serving-cert" not found
Jul 09 11:39:42.043593 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:42.043557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:42.965948 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:42.965384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5664ec9e-5993-4231-a028-2b5b22d93682-maas-api-tls\") pod \"maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb\" (UID: \"5664ec9e-5993-4231-a028-2b5b22d93682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:42.971557 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:42.971526    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5664ec9e-5993-4231-a028-2b5b22d93682-maas-api-tls\") pod \"maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb\" (UID: \"5664ec9e-5993-4231-a028-2b5b22d93682\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:43.038570 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:43.038530    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:39:43.038678 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:43.038613    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:39:43.038678 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:43.038652    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:39:43.082083 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:43.082059    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:43.230857 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:43.230820    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"]
Jul 09 11:39:43.233337 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:39:43.233299    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5664ec9e_5993_4231_a028_2b5b22d93682.slice/crio-db2e943b8369c0d4d6ece1d0ddd2f1e73ebc750cb0b67d194f0693ec9c1ecb97 WatchSource:0}: Error finding container db2e943b8369c0d4d6ece1d0ddd2f1e73ebc750cb0b67d194f0693ec9c1ecb97: Status 404 returned error can't find the container with id db2e943b8369c0d4d6ece1d0ddd2f1e73ebc750cb0b67d194f0693ec9c1ecb97
Jul 09 11:39:43.529047 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:43.529013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:43.731944 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:43.731906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw" event={"ID":"56faf4bb-af96-4417-8a53-849a05d4367e","Type":"ContainerStarted","Data":"539d84b7b4a275c23bda49df8b3c76a288df953d37a2bba2c85f719ff6b5d100"}
Jul 09 11:39:43.733027 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:43.733001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb" event={"ID":"5664ec9e-5993-4231-a028-2b5b22d93682","Type":"ContainerStarted","Data":"db2e943b8369c0d4d6ece1d0ddd2f1e73ebc750cb0b67d194f0693ec9c1ecb97"}
Jul 09 11:39:43.756934 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:43.756880    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw" podStartSLOduration=2.102754354 podStartE2EDuration="4.756864717s" totalImagesPullingTime="2.654110363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:39:43.755262716 +0000 UTC m=+1667.346993881" watchObservedRunningTime="2026-07-09 11:39:43.756864717 +0000 UTC m=+1667.348595880"
Jul 09 11:39:43.869295 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:43.869263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:44.225827 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:44.225716    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:44.230687 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:44.230662    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:44.738873 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:44.738838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb" event={"ID":"5664ec9e-5993-4231-a028-2b5b22d93682","Type":"ContainerStarted","Data":"ee33787c832b4b820339796df2f83dc2d79785b467ee1662cd374817d1e918b3"}
Jul 09 11:39:44.739035 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:44.738932    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:44.739308 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:44.739290    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:44.740325 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:44.740309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:44.758360 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:44.758318    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb" podStartSLOduration=3.1708593880000002 podStartE2EDuration="3.758302554s" totalImagesPullingTime="587.443166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:39:44.757329707 +0000 UTC m=+1668.349060873" watchObservedRunningTime="2026-07-09 11:39:44.758302554 +0000 UTC m=+1668.350033719"
Jul 09 11:39:44.926514 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:44.926481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:45.024617 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:45.024581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:46.165698 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:46.165663    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:46.315331 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:46.315300    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"]
Jul 09 11:39:46.333168 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:46.333133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:46.435507 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:46.435427    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:46.540578 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:46.540548    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"]
Jul 09 11:39:46.571347 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:46.571287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:46.748315 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:46.748219    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb" podUID="5664ec9e-5993-4231-a028-2b5b22d93682" containerName="maas-api" containerID="cri-o://ee33787c832b4b820339796df2f83dc2d79785b467ee1662cd374817d1e918b3" gracePeriod=30
Jul 09 11:39:46.748315 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:46.748271    2563 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw" secret="" err="secret \"e2e-ait-5bca60b8-openshift-default-dockercfg-dpb7x\" not found"
Jul 09 11:39:47.001070 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.001051    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:47.101641 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.101605    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5664ec9e-5993-4231-a028-2b5b22d93682-openshift-service-ca\") pod \"5664ec9e-5993-4231-a028-2b5b22d93682\" (UID: \"5664ec9e-5993-4231-a028-2b5b22d93682\") "
Jul 09 11:39:47.101814 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.101681    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5664ec9e-5993-4231-a028-2b5b22d93682-maas-api-tls\") pod \"5664ec9e-5993-4231-a028-2b5b22d93682\" (UID: \"5664ec9e-5993-4231-a028-2b5b22d93682\") "
Jul 09 11:39:47.101814 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.101749    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhdnx\" (UniqueName: \"kubernetes.io/projected/5664ec9e-5993-4231-a028-2b5b22d93682-kube-api-access-nhdnx\") pod \"5664ec9e-5993-4231-a028-2b5b22d93682\" (UID: \"5664ec9e-5993-4231-a028-2b5b22d93682\") "
Jul 09 11:39:47.102005 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.101985    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5664ec9e-5993-4231-a028-2b5b22d93682-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5664ec9e-5993-4231-a028-2b5b22d93682" (UID: "5664ec9e-5993-4231-a028-2b5b22d93682"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:39:47.102075 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.102058    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5664ec9e-5993-4231-a028-2b5b22d93682-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:39:47.103940 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.103901    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5664ec9e-5993-4231-a028-2b5b22d93682-kube-api-access-nhdnx" (OuterVolumeSpecName: "kube-api-access-nhdnx") pod "5664ec9e-5993-4231-a028-2b5b22d93682" (UID: "5664ec9e-5993-4231-a028-2b5b22d93682"). InnerVolumeSpecName "kube-api-access-nhdnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:39:47.103940 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.103911    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5664ec9e-5993-4231-a028-2b5b22d93682-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5664ec9e-5993-4231-a028-2b5b22d93682" (UID: "5664ec9e-5993-4231-a028-2b5b22d93682"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 11:39:47.202758 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.202722    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5664ec9e-5993-4231-a028-2b5b22d93682-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:39:47.202758 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.202753    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhdnx\" (UniqueName: \"kubernetes.io/projected/5664ec9e-5993-4231-a028-2b5b22d93682-kube-api-access-nhdnx\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:39:47.753560 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.753524    2563 generic.go:356] "Generic (PLEG): container finished" podID="5664ec9e-5993-4231-a028-2b5b22d93682" containerID="ee33787c832b4b820339796df2f83dc2d79785b467ee1662cd374817d1e918b3" exitCode=0
Jul 09 11:39:47.753729 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.753567    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"
Jul 09 11:39:47.753729 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.753607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb" event={"ID":"5664ec9e-5993-4231-a028-2b5b22d93682","Type":"ContainerDied","Data":"ee33787c832b4b820339796df2f83dc2d79785b467ee1662cd374817d1e918b3"}
Jul 09 11:39:47.753729 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.753681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb" event={"ID":"5664ec9e-5993-4231-a028-2b5b22d93682","Type":"ContainerDied","Data":"db2e943b8369c0d4d6ece1d0ddd2f1e73ebc750cb0b67d194f0693ec9c1ecb97"}
Jul 09 11:39:47.753729 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.753703    2563 scope.go:117] "RemoveContainer" containerID="ee33787c832b4b820339796df2f83dc2d79785b467ee1662cd374817d1e918b3"
Jul 09 11:39:47.753959 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.753905    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw" podUID="56faf4bb-af96-4417-8a53-849a05d4367e" containerName="istio-proxy" containerID="cri-o://539d84b7b4a275c23bda49df8b3c76a288df953d37a2bba2c85f719ff6b5d100" gracePeriod=30
Jul 09 11:39:47.762051 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.762036    2563 scope.go:117] "RemoveContainer" containerID="ee33787c832b4b820339796df2f83dc2d79785b467ee1662cd374817d1e918b3"
Jul 09 11:39:47.762281 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:39:47.762264    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee33787c832b4b820339796df2f83dc2d79785b467ee1662cd374817d1e918b3\": container with ID starting with ee33787c832b4b820339796df2f83dc2d79785b467ee1662cd374817d1e918b3 not found: ID does not exist" containerID="ee33787c832b4b820339796df2f83dc2d79785b467ee1662cd374817d1e918b3"
Jul 09 11:39:47.762345 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.762292    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee33787c832b4b820339796df2f83dc2d79785b467ee1662cd374817d1e918b3"} err="failed to get container status \"ee33787c832b4b820339796df2f83dc2d79785b467ee1662cd374817d1e918b3\": rpc error: code = NotFound desc = could not find container \"ee33787c832b4b820339796df2f83dc2d79785b467ee1662cd374817d1e918b3\": container with ID starting with ee33787c832b4b820339796df2f83dc2d79785b467ee1662cd374817d1e918b3 not found: ID does not exist"
Jul 09 11:39:47.792131 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.792101    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"]
Jul 09 11:39:47.797049 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.797028    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5bca60b8-5b864c4b68-nxgkb"]
Jul 09 11:39:47.840919 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:47.840895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:48.950641 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:48.950611    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5664ec9e-5993-4231-a028-2b5b22d93682" path="/var/lib/kubelet/pods/5664ec9e-5993-4231-a028-2b5b22d93682/volumes"
Jul 09 11:39:52.565578 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.565540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:52.733257 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.733220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:52.814080 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.814041    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"]
Jul 09 11:39:52.814532 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.814512    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5664ec9e-5993-4231-a028-2b5b22d93682" containerName="maas-api"
Jul 09 11:39:52.814582 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.814536    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5664ec9e-5993-4231-a028-2b5b22d93682" containerName="maas-api"
Jul 09 11:39:52.814655 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.814643    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5664ec9e-5993-4231-a028-2b5b22d93682" containerName="maas-api"
Jul 09 11:39:52.818216 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.818154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:52.821384 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.821360    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ca6bd9c3-openshift-default-dockercfg-tz8x2\""
Jul 09 11:39:52.837910 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.837880    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"]
Jul 09 11:39:52.878693 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.878658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:52.954802 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.954760    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-credential-socket\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:52.954960 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.954814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-podinfo\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:52.954960 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.954865    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-workload-certs\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:52.954960 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.954888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-workload-socket\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:52.954960 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.954910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istiod-ca-cert\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:52.955086 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.954995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-envoy\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:52.955086 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.955074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-token\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:52.955149 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.955115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkv8b\" (UniqueName: \"kubernetes.io/projected/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-kube-api-access-fkv8b\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:52.955183 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:52.955172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-data\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.009955 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.009925    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:53.050067 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.050031    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:53.056116 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.056080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-token\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.056317 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.056132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkv8b\" (UniqueName: \"kubernetes.io/projected/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-kube-api-access-fkv8b\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.056317 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.056161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-data\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.056317 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.056198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-credential-socket\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.056317 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.056228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-podinfo\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.056317 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.056267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-workload-certs\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.056317 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.056299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-workload-socket\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.056621 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.056447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istiod-ca-cert\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.056621 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.056508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-envoy\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.056723 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.056652    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-credential-socket\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.056723 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.056687    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-workload-certs\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.056843 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.056743    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-workload-socket\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.056843 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.056827    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-data\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.057153 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.057135    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istiod-ca-cert\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.058601 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.058572    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-envoy\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.058968 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.058944    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-podinfo\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.067044 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.067024    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-token\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.067257 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.067236    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkv8b\" (UniqueName: \"kubernetes.io/projected/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-kube-api-access-fkv8b\") pod \"e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") " pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.146938 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.146843    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:53.157510 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.157480    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-workload-certs\") pod \"56faf4bb-af96-4417-8a53-849a05d4367e\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") "
Jul 09 11:39:53.157662 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.157525    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56faf4bb-af96-4417-8a53-849a05d4367e-istio-token\") pod \"56faf4bb-af96-4417-8a53-849a05d4367e\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") "
Jul 09 11:39:53.157662 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.157576    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56faf4bb-af96-4417-8a53-849a05d4367e-istiod-ca-cert\") pod \"56faf4bb-af96-4417-8a53-849a05d4367e\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") "
Jul 09 11:39:53.157662 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.157605    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dp7z\" (UniqueName: \"kubernetes.io/projected/56faf4bb-af96-4417-8a53-849a05d4367e-kube-api-access-8dp7z\") pod \"56faf4bb-af96-4417-8a53-849a05d4367e\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") "
Jul 09 11:39:53.157662 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.157643    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-istio-envoy\") pod \"56faf4bb-af96-4417-8a53-849a05d4367e\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") "
Jul 09 11:39:53.157881 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.157678    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-credential-socket\") pod \"56faf4bb-af96-4417-8a53-849a05d4367e\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") "
Jul 09 11:39:53.157881 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.157730    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56faf4bb-af96-4417-8a53-849a05d4367e-istio-podinfo\") pod \"56faf4bb-af96-4417-8a53-849a05d4367e\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") "
Jul 09 11:39:53.157881 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.157769    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-workload-socket\") pod \"56faf4bb-af96-4417-8a53-849a05d4367e\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") "
Jul 09 11:39:53.157881 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.157820    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-istio-data\") pod \"56faf4bb-af96-4417-8a53-849a05d4367e\" (UID: \"56faf4bb-af96-4417-8a53-849a05d4367e\") "
Jul 09 11:39:53.158081 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.157765    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "56faf4bb-af96-4417-8a53-849a05d4367e" (UID: "56faf4bb-af96-4417-8a53-849a05d4367e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:39:53.158081 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.157954    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "56faf4bb-af96-4417-8a53-849a05d4367e" (UID: "56faf4bb-af96-4417-8a53-849a05d4367e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:39:53.158182 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.158068    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "56faf4bb-af96-4417-8a53-849a05d4367e" (UID: "56faf4bb-af96-4417-8a53-849a05d4367e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:39:53.158231 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.158194    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:39:53.158231 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.158216    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:39:53.158312 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.158231    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:39:53.158312 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.158279    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56faf4bb-af96-4417-8a53-849a05d4367e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "56faf4bb-af96-4417-8a53-849a05d4367e" (UID: "56faf4bb-af96-4417-8a53-849a05d4367e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:39:53.159988 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.159963    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56faf4bb-af96-4417-8a53-849a05d4367e-kube-api-access-8dp7z" (OuterVolumeSpecName: "kube-api-access-8dp7z") pod "56faf4bb-af96-4417-8a53-849a05d4367e" (UID: "56faf4bb-af96-4417-8a53-849a05d4367e"). InnerVolumeSpecName "kube-api-access-8dp7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:39:53.159988 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.159989    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56faf4bb-af96-4417-8a53-849a05d4367e-istio-token" (OuterVolumeSpecName: "istio-token") pod "56faf4bb-af96-4417-8a53-849a05d4367e" (UID: "56faf4bb-af96-4417-8a53-849a05d4367e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:39:53.160952 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.160932    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-istio-data" (OuterVolumeSpecName: "istio-data") pod "56faf4bb-af96-4417-8a53-849a05d4367e" (UID: "56faf4bb-af96-4417-8a53-849a05d4367e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:39:53.162538 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.162508    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56faf4bb-af96-4417-8a53-849a05d4367e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "56faf4bb-af96-4417-8a53-849a05d4367e" (UID: "56faf4bb-af96-4417-8a53-849a05d4367e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 11:39:53.162634 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.162585    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "56faf4bb-af96-4417-8a53-849a05d4367e" (UID: "56faf4bb-af96-4417-8a53-849a05d4367e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:39:53.259743 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.259714    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:39:53.259743 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.259741    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56faf4bb-af96-4417-8a53-849a05d4367e-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:39:53.259743 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.259750    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56faf4bb-af96-4417-8a53-849a05d4367e-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:39:53.259975 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.259758    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dp7z\" (UniqueName: \"kubernetes.io/projected/56faf4bb-af96-4417-8a53-849a05d4367e-kube-api-access-8dp7z\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:39:53.259975 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.259771    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56faf4bb-af96-4417-8a53-849a05d4367e-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:39:53.259975 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.259802    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56faf4bb-af96-4417-8a53-849a05d4367e-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:39:53.289648 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.289622    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"]
Jul 09 11:39:53.291620 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:39:53.291594    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a218a8_ce05_4c4a_afe7_a50ae92a45cd.slice/crio-dd88ffa49c99db16f702671ed93c9635d44dea236ea5d85573a7cd132bd2fd0a WatchSource:0}: Error finding container dd88ffa49c99db16f702671ed93c9635d44dea236ea5d85573a7cd132bd2fd0a: Status 404 returned error can't find the container with id dd88ffa49c99db16f702671ed93c9635d44dea236ea5d85573a7cd132bd2fd0a
Jul 09 11:39:53.294460 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.294432    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:39:53.294539 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.294496    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:39:53.294539 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.294523    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:39:53.781654 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.781622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw" event={"ID":"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd","Type":"ContainerStarted","Data":"c2f8fd72d5a709f514ccb50bddcff2d9828eb5a1f02d9da2dc145f54e3aeb00f"}
Jul 09 11:39:53.782072 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.781672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw" event={"ID":"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd","Type":"ContainerStarted","Data":"dd88ffa49c99db16f702671ed93c9635d44dea236ea5d85573a7cd132bd2fd0a"}
Jul 09 11:39:53.782915 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.782894    2563 generic.go:356] "Generic (PLEG): container finished" podID="56faf4bb-af96-4417-8a53-849a05d4367e" containerID="539d84b7b4a275c23bda49df8b3c76a288df953d37a2bba2c85f719ff6b5d100" exitCode=0
Jul 09 11:39:53.783007 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.782934    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"
Jul 09 11:39:53.783007 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.782956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw" event={"ID":"56faf4bb-af96-4417-8a53-849a05d4367e","Type":"ContainerDied","Data":"539d84b7b4a275c23bda49df8b3c76a288df953d37a2bba2c85f719ff6b5d100"}
Jul 09 11:39:53.783007 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.782990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw" event={"ID":"56faf4bb-af96-4417-8a53-849a05d4367e","Type":"ContainerDied","Data":"f0037bc70b5b18eabd2d2e5f0f74a363fe36bf09e7e24f430ebde2364b22ab18"}
Jul 09 11:39:53.783110 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.783011    2563 scope.go:117] "RemoveContainer" containerID="539d84b7b4a275c23bda49df8b3c76a288df953d37a2bba2c85f719ff6b5d100"
Jul 09 11:39:53.791730 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.791714    2563 scope.go:117] "RemoveContainer" containerID="539d84b7b4a275c23bda49df8b3c76a288df953d37a2bba2c85f719ff6b5d100"
Jul 09 11:39:53.792000 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:39:53.791980    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"539d84b7b4a275c23bda49df8b3c76a288df953d37a2bba2c85f719ff6b5d100\": container with ID starting with 539d84b7b4a275c23bda49df8b3c76a288df953d37a2bba2c85f719ff6b5d100 not found: ID does not exist" containerID="539d84b7b4a275c23bda49df8b3c76a288df953d37a2bba2c85f719ff6b5d100"
Jul 09 11:39:53.792061 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.792010    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"539d84b7b4a275c23bda49df8b3c76a288df953d37a2bba2c85f719ff6b5d100"} err="failed to get container status \"539d84b7b4a275c23bda49df8b3c76a288df953d37a2bba2c85f719ff6b5d100\": rpc error: code = NotFound desc = could not find container \"539d84b7b4a275c23bda49df8b3c76a288df953d37a2bba2c85f719ff6b5d100\": container with ID starting with 539d84b7b4a275c23bda49df8b3c76a288df953d37a2bba2c85f719ff6b5d100 not found: ID does not exist"
Jul 09 11:39:53.805389 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.805348    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw" podStartSLOduration=1.8053335050000001 podStartE2EDuration="1.805333505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:39:53.804249773 +0000 UTC m=+1677.395980938" watchObservedRunningTime="2026-07-09 11:39:53.805333505 +0000 UTC m=+1677.397064734"
Jul 09 11:39:53.819403 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.819380    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"]
Jul 09 11:39:53.825092 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.825073    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5bca60b8-openshift-default-6b7ff7ddb5-zx4jw"]
Jul 09 11:39:53.962397 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:53.962362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:54.060362 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.060259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:54.110554 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.110521    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"]
Jul 09 11:39:54.110911 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.110895    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56faf4bb-af96-4417-8a53-849a05d4367e" containerName="istio-proxy"
Jul 09 11:39:54.111002 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.110914    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="56faf4bb-af96-4417-8a53-849a05d4367e" containerName="istio-proxy"
Jul 09 11:39:54.111055 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.111021    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="56faf4bb-af96-4417-8a53-849a05d4367e" containerName="istio-proxy"
Jul 09 11:39:54.115228 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.115202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"
Jul 09 11:39:54.117738 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.117717    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ca6bd9c3-serving-cert\""
Jul 09 11:39:54.135969 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.135942    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"]
Jul 09 11:39:54.147357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.147325    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:54.152851 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.152828    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:54.161203 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.161155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:54.268927 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.268877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87e0e901-ba8e-44de-9858-1546fd779c23-maas-api-tls\") pod \"maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb\" (UID: \"87e0e901-ba8e-44de-9858-1546fd779c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"
Jul 09 11:39:54.269110 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.269011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e0e901-ba8e-44de-9858-1546fd779c23-openshift-service-ca\") pod \"maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb\" (UID: \"87e0e901-ba8e-44de-9858-1546fd779c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"
Jul 09 11:39:54.269110 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.269067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5x8d\" (UniqueName: \"kubernetes.io/projected/87e0e901-ba8e-44de-9858-1546fd779c23-kube-api-access-h5x8d\") pod \"maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb\" (UID: \"87e0e901-ba8e-44de-9858-1546fd779c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"
Jul 09 11:39:54.369982 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.369888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87e0e901-ba8e-44de-9858-1546fd779c23-maas-api-tls\") pod \"maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb\" (UID: \"87e0e901-ba8e-44de-9858-1546fd779c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"
Jul 09 11:39:54.370144 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.369993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e0e901-ba8e-44de-9858-1546fd779c23-openshift-service-ca\") pod \"maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb\" (UID: \"87e0e901-ba8e-44de-9858-1546fd779c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"
Jul 09 11:39:54.370144 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:39:54.370041    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-serving-cert: secret "maas-api-e2e-ait-ca6bd9c3-serving-cert" not found
Jul 09 11:39:54.370144 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:39:54.370108    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87e0e901-ba8e-44de-9858-1546fd779c23-maas-api-tls podName:87e0e901-ba8e-44de-9858-1546fd779c23 nodeName:}" failed. No retries permitted until 2026-07-09 11:39:54.870092226 +0000 UTC m=+1678.461823367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87e0e901-ba8e-44de-9858-1546fd779c23-maas-api-tls") pod "maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb" (UID: "87e0e901-ba8e-44de-9858-1546fd779c23") : secret "maas-api-e2e-ait-ca6bd9c3-serving-cert" not found
Jul 09 11:39:54.370144 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.370044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5x8d\" (UniqueName: \"kubernetes.io/projected/87e0e901-ba8e-44de-9858-1546fd779c23-kube-api-access-h5x8d\") pod \"maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb\" (UID: \"87e0e901-ba8e-44de-9858-1546fd779c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"
Jul 09 11:39:54.370593 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.370573    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e0e901-ba8e-44de-9858-1546fd779c23-openshift-service-ca\") pod \"maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb\" (UID: \"87e0e901-ba8e-44de-9858-1546fd779c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"
Jul 09 11:39:54.383351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.383323    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5x8d\" (UniqueName: \"kubernetes.io/projected/87e0e901-ba8e-44de-9858-1546fd779c23-kube-api-access-h5x8d\") pod \"maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb\" (UID: \"87e0e901-ba8e-44de-9858-1546fd779c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"
Jul 09 11:39:54.542111 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.542073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:54.788433 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.788396    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:54.789506 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.789488    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:39:54.792961 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.792926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:54.875179 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.875145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87e0e901-ba8e-44de-9858-1546fd779c23-maas-api-tls\") pod \"maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb\" (UID: \"87e0e901-ba8e-44de-9858-1546fd779c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"
Jul 09 11:39:54.877814 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.877766    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87e0e901-ba8e-44de-9858-1546fd779c23-maas-api-tls\") pod \"maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb\" (UID: \"87e0e901-ba8e-44de-9858-1546fd779c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"
Jul 09 11:39:54.887896 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.887869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:54.951738 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:54.951699    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56faf4bb-af96-4417-8a53-849a05d4367e" path="/var/lib/kubelet/pods/56faf4bb-af96-4417-8a53-849a05d4367e/volumes"
Jul 09 11:39:55.027187 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:55.027145    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"
Jul 09 11:39:55.202933 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:55.202909    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"]
Jul 09 11:39:55.204309 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:39:55.204282    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e0e901_ba8e_44de_9858_1546fd779c23.slice/crio-b2e98dfcdd4e01617632c048d6f85f4469c07075f760e6c190a52562e47c1f32 WatchSource:0}: Error finding container b2e98dfcdd4e01617632c048d6f85f4469c07075f760e6c190a52562e47c1f32: Status 404 returned error can't find the container with id b2e98dfcdd4e01617632c048d6f85f4469c07075f760e6c190a52562e47c1f32
Jul 09 11:39:55.794335 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:55.794297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb" event={"ID":"87e0e901-ba8e-44de-9858-1546fd779c23","Type":"ContainerStarted","Data":"1e53273d32094ab385523f003720c90f212caad8386bb3611a2a948a7ec7e45f"}
Jul 09 11:39:55.794839 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:55.794349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb" event={"ID":"87e0e901-ba8e-44de-9858-1546fd779c23","Type":"ContainerStarted","Data":"b2e98dfcdd4e01617632c048d6f85f4469c07075f760e6c190a52562e47c1f32"}
Jul 09 11:39:55.813765 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:55.813717    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb" podStartSLOduration=1.365720544 podStartE2EDuration="1.813701577s" totalImagesPullingTime="447.981033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:39:55.812534841 +0000 UTC m=+1679.404266006" watchObservedRunningTime="2026-07-09 11:39:55.813701577 +0000 UTC m=+1679.405432740"
Jul 09 11:39:56.071181 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:56.071153    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:56.798277 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:56.798246    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"
Jul 09 11:39:59.007200 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.007164    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:59.409249 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.409154    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"]
Jul 09 11:39:59.409434 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.409407    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb" podUID="87e0e901-ba8e-44de-9858-1546fd779c23" containerName="maas-api" containerID="cri-o://1e53273d32094ab385523f003720c90f212caad8386bb3611a2a948a7ec7e45f" gracePeriod=30
Jul 09 11:39:59.437132 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.437102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:39:59.687887 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.687858    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"
Jul 09 11:39:59.811380 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.811345    2563 generic.go:356] "Generic (PLEG): container finished" podID="87e0e901-ba8e-44de-9858-1546fd779c23" containerID="1e53273d32094ab385523f003720c90f212caad8386bb3611a2a948a7ec7e45f" exitCode=0
Jul 09 11:39:59.811517 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.811392    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"
Jul 09 11:39:59.811517 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.811393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb" event={"ID":"87e0e901-ba8e-44de-9858-1546fd779c23","Type":"ContainerDied","Data":"1e53273d32094ab385523f003720c90f212caad8386bb3611a2a948a7ec7e45f"}
Jul 09 11:39:59.811517 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.811511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb" event={"ID":"87e0e901-ba8e-44de-9858-1546fd779c23","Type":"ContainerDied","Data":"b2e98dfcdd4e01617632c048d6f85f4469c07075f760e6c190a52562e47c1f32"}
Jul 09 11:39:59.811651 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.811539    2563 scope.go:117] "RemoveContainer" containerID="1e53273d32094ab385523f003720c90f212caad8386bb3611a2a948a7ec7e45f"
Jul 09 11:39:59.819509 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.819488    2563 scope.go:117] "RemoveContainer" containerID="1e53273d32094ab385523f003720c90f212caad8386bb3611a2a948a7ec7e45f"
Jul 09 11:39:59.819764 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:39:59.819745    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e53273d32094ab385523f003720c90f212caad8386bb3611a2a948a7ec7e45f\": container with ID starting with 1e53273d32094ab385523f003720c90f212caad8386bb3611a2a948a7ec7e45f not found: ID does not exist" containerID="1e53273d32094ab385523f003720c90f212caad8386bb3611a2a948a7ec7e45f"
Jul 09 11:39:59.819831 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.819772    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e53273d32094ab385523f003720c90f212caad8386bb3611a2a948a7ec7e45f"} err="failed to get container status \"1e53273d32094ab385523f003720c90f212caad8386bb3611a2a948a7ec7e45f\": rpc error: code = NotFound desc = could not find container \"1e53273d32094ab385523f003720c90f212caad8386bb3611a2a948a7ec7e45f\": container with ID starting with 1e53273d32094ab385523f003720c90f212caad8386bb3611a2a948a7ec7e45f not found: ID does not exist"
Jul 09 11:39:59.821076 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.821060    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87e0e901-ba8e-44de-9858-1546fd779c23-maas-api-tls\") pod \"87e0e901-ba8e-44de-9858-1546fd779c23\" (UID: \"87e0e901-ba8e-44de-9858-1546fd779c23\") "
Jul 09 11:39:59.821123 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.821095    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e0e901-ba8e-44de-9858-1546fd779c23-openshift-service-ca\") pod \"87e0e901-ba8e-44de-9858-1546fd779c23\" (UID: \"87e0e901-ba8e-44de-9858-1546fd779c23\") "
Jul 09 11:39:59.821168 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.821160    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5x8d\" (UniqueName: \"kubernetes.io/projected/87e0e901-ba8e-44de-9858-1546fd779c23-kube-api-access-h5x8d\") pod \"87e0e901-ba8e-44de-9858-1546fd779c23\" (UID: \"87e0e901-ba8e-44de-9858-1546fd779c23\") "
Jul 09 11:39:59.821502 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.821478    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e0e901-ba8e-44de-9858-1546fd779c23-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "87e0e901-ba8e-44de-9858-1546fd779c23" (UID: "87e0e901-ba8e-44de-9858-1546fd779c23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:39:59.822924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.822899    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e0e901-ba8e-44de-9858-1546fd779c23-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "87e0e901-ba8e-44de-9858-1546fd779c23" (UID: "87e0e901-ba8e-44de-9858-1546fd779c23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 11:39:59.823201 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.823188    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e0e901-ba8e-44de-9858-1546fd779c23-kube-api-access-h5x8d" (OuterVolumeSpecName: "kube-api-access-h5x8d") pod "87e0e901-ba8e-44de-9858-1546fd779c23" (UID: "87e0e901-ba8e-44de-9858-1546fd779c23"). InnerVolumeSpecName "kube-api-access-h5x8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:39:59.922659 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.922572    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87e0e901-ba8e-44de-9858-1546fd779c23-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:39:59.922659 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.922605    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e0e901-ba8e-44de-9858-1546fd779c23-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:39:59.922659 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:39:59.922615    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5x8d\" (UniqueName: \"kubernetes.io/projected/87e0e901-ba8e-44de-9858-1546fd779c23-kube-api-access-h5x8d\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:00.134042 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:00.134013    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"]
Jul 09 11:40:00.139356 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:00.139317    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca6bd9c3-5446d4db54-xlndb"]
Jul 09 11:40:00.953058 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:00.953017    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87e0e901-ba8e-44de-9858-1546fd779c23" path="/var/lib/kubelet/pods/87e0e901-ba8e-44de-9858-1546fd779c23/volumes"
Jul 09 11:40:04.836060 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:04.836027    2563 generic.go:356] "Generic (PLEG): container finished" podID="f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" containerID="c2f8fd72d5a709f514ccb50bddcff2d9828eb5a1f02d9da2dc145f54e3aeb00f" exitCode=0
Jul 09 11:40:04.836486 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:04.836104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw" event={"ID":"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd","Type":"ContainerDied","Data":"c2f8fd72d5a709f514ccb50bddcff2d9828eb5a1f02d9da2dc145f54e3aeb00f"}
Jul 09 11:40:04.836533 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:04.836525    2563 scope.go:117] "RemoveContainer" containerID="c2f8fd72d5a709f514ccb50bddcff2d9828eb5a1f02d9da2dc145f54e3aeb00f"
Jul 09 11:40:04.837467 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:04.837444    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:40:04.837524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:04.837511    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:40:04.837559 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:04.837547    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:40:05.097180 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:05.097142    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"]
Jul 09 11:40:05.119225 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:05.119190    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:05.241146 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:05.241111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:05.843134 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:05.843101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw" event={"ID":"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd","Type":"ContainerStarted","Data":"a4a63645bdc096c4d6dac0648fdfdefdf5fe694a053aff617f68789fb511db48"}
Jul 09 11:40:05.843544 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:05.843213    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw" podUID="f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" containerName="istio-proxy" containerID="cri-o://a4a63645bdc096c4d6dac0648fdfdefdf5fe694a053aff617f68789fb511db48" gracePeriod=30
Jul 09 11:40:09.788573 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:09.788536    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:40:11.089700 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.089677    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:40:11.127636 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.127556    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-workload-socket\") pod \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") "
Jul 09 11:40:11.127636 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.127602    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-token\") pod \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") "
Jul 09 11:40:11.127636 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.127623    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-podinfo\") pod \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") "
Jul 09 11:40:11.127945 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.127660    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istiod-ca-cert\") pod \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") "
Jul 09 11:40:11.127945 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.127751    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-envoy\") pod \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") "
Jul 09 11:40:11.127945 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.127802    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-data\") pod \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") "
Jul 09 11:40:11.127945 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.127829    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkv8b\" (UniqueName: \"kubernetes.io/projected/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-kube-api-access-fkv8b\") pod \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") "
Jul 09 11:40:11.127945 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.127886    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-credential-socket\") pod \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") "
Jul 09 11:40:11.127945 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.127886    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" (UID: "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:40:11.127945 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.127934    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-workload-certs\") pod \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\" (UID: \"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd\") "
Jul 09 11:40:11.128296 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.128214    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:11.128620 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.128414    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" (UID: "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:40:11.128620 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.128563    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" (UID: "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:40:11.128620 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.128563    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" (UID: "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:40:11.128903 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.128878    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-data" (OuterVolumeSpecName: "istio-data") pod "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" (UID: "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:40:11.130062 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.130016    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" (UID: "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:40:11.130155 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.130081    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" (UID: "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 11:40:11.130216 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.130204    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-token" (OuterVolumeSpecName: "istio-token") pod "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" (UID: "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:40:11.130427 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.130400    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-kube-api-access-fkv8b" (OuterVolumeSpecName: "kube-api-access-fkv8b") pod "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" (UID: "f4a218a8-ce05-4c4a-afe7-a50ae92a45cd"). InnerVolumeSpecName "kube-api-access-fkv8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:40:11.229346 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.229311    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:11.229346 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.229341    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:11.229346 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.229349    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:11.229561 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.229358    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkv8b\" (UniqueName: \"kubernetes.io/projected/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-kube-api-access-fkv8b\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:11.229561 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.229368    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:11.229561 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.229377    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:11.229561 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.229384    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:11.229561 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.229392    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:11.392071 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.391964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:11.488993 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.488961    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"]
Jul 09 11:40:11.489339 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.489323    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e0e901-ba8e-44de-9858-1546fd779c23" containerName="maas-api"
Jul 09 11:40:11.489480 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.489343    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e0e901-ba8e-44de-9858-1546fd779c23" containerName="maas-api"
Jul 09 11:40:11.489480 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.489368    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" containerName="istio-proxy"
Jul 09 11:40:11.489480 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.489376    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" containerName="istio-proxy"
Jul 09 11:40:11.489480 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.489468    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" containerName="istio-proxy"
Jul 09 11:40:11.489480 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.489481    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" containerName="istio-proxy"
Jul 09 11:40:11.489712 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.489490    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e0e901-ba8e-44de-9858-1546fd779c23" containerName="maas-api"
Jul 09 11:40:11.495360 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.495324    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:11.524159 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.524112    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"]
Jul 09 11:40:11.524333 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.524243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.528389 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.528365    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-73012e4a-openshift-default-dockercfg-847f4\""
Jul 09 11:40:11.583541 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.583505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:11.632902 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.632860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-envoy\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.632902 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.632908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-podinfo\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.633161 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.632977    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn9fx\" (UniqueName: \"kubernetes.io/projected/2c6f0c42-1d7b-49b2-b915-c64f7450f962-kube-api-access-xn9fx\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.633161 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.633022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istiod-ca-cert\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.633161 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.633059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-workload-certs\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.633161 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.633080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-data\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.633161 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.633133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-workload-socket\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.633161 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.633156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-token\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.633381 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.633186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-credential-socket\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.734269 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.734173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-envoy\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.734269 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.734218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-podinfo\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.734269 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.734254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn9fx\" (UniqueName: \"kubernetes.io/projected/2c6f0c42-1d7b-49b2-b915-c64f7450f962-kube-api-access-xn9fx\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.734269 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.734276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istiod-ca-cert\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.734605 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.734299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-workload-certs\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.734605 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.734322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-data\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.734605 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.734359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-workload-socket\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.734605 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.734382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-token\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.734605 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.734408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-credential-socket\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.734927 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.734735    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-workload-certs\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.734927 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.734844    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-data\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.734927 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.734878    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-workload-socket\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.735079 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.735016    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-credential-socket\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.735079 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.735054    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istiod-ca-cert\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.736576 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.736553    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-envoy\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.736810 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.736776    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-podinfo\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.743039 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.743016    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn9fx\" (UniqueName: \"kubernetes.io/projected/2c6f0c42-1d7b-49b2-b915-c64f7450f962-kube-api-access-xn9fx\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.743146 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.743129    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-token\") pod \"e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") " pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.835079 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.835044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:11.872107 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.872071    2563 generic.go:356] "Generic (PLEG): container finished" podID="f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" containerID="a4a63645bdc096c4d6dac0648fdfdefdf5fe694a053aff617f68789fb511db48" exitCode=0
Jul 09 11:40:11.872260 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.872124    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"
Jul 09 11:40:11.872260 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.872161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw" event={"ID":"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd","Type":"ContainerDied","Data":"a4a63645bdc096c4d6dac0648fdfdefdf5fe694a053aff617f68789fb511db48"}
Jul 09 11:40:11.872260 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.872230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw" event={"ID":"f4a218a8-ce05-4c4a-afe7-a50ae92a45cd","Type":"ContainerDied","Data":"dd88ffa49c99db16f702671ed93c9635d44dea236ea5d85573a7cd132bd2fd0a"}
Jul 09 11:40:11.872367 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.872264    2563 scope.go:117] "RemoveContainer" containerID="a4a63645bdc096c4d6dac0648fdfdefdf5fe694a053aff617f68789fb511db48"
Jul 09 11:40:11.881599 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.881574    2563 scope.go:117] "RemoveContainer" containerID="c2f8fd72d5a709f514ccb50bddcff2d9828eb5a1f02d9da2dc145f54e3aeb00f"
Jul 09 11:40:11.891953 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.891931    2563 scope.go:117] "RemoveContainer" containerID="a4a63645bdc096c4d6dac0648fdfdefdf5fe694a053aff617f68789fb511db48"
Jul 09 11:40:11.892321 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:40:11.892292    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4a63645bdc096c4d6dac0648fdfdefdf5fe694a053aff617f68789fb511db48\": container with ID starting with a4a63645bdc096c4d6dac0648fdfdefdf5fe694a053aff617f68789fb511db48 not found: ID does not exist" containerID="a4a63645bdc096c4d6dac0648fdfdefdf5fe694a053aff617f68789fb511db48"
Jul 09 11:40:11.892402 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.892360    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4a63645bdc096c4d6dac0648fdfdefdf5fe694a053aff617f68789fb511db48"} err="failed to get container status \"a4a63645bdc096c4d6dac0648fdfdefdf5fe694a053aff617f68789fb511db48\": rpc error: code = NotFound desc = could not find container \"a4a63645bdc096c4d6dac0648fdfdefdf5fe694a053aff617f68789fb511db48\": container with ID starting with a4a63645bdc096c4d6dac0648fdfdefdf5fe694a053aff617f68789fb511db48 not found: ID does not exist"
Jul 09 11:40:11.892402 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.892386    2563 scope.go:117] "RemoveContainer" containerID="c2f8fd72d5a709f514ccb50bddcff2d9828eb5a1f02d9da2dc145f54e3aeb00f"
Jul 09 11:40:11.892690 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:40:11.892668    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f8fd72d5a709f514ccb50bddcff2d9828eb5a1f02d9da2dc145f54e3aeb00f\": container with ID starting with c2f8fd72d5a709f514ccb50bddcff2d9828eb5a1f02d9da2dc145f54e3aeb00f not found: ID does not exist" containerID="c2f8fd72d5a709f514ccb50bddcff2d9828eb5a1f02d9da2dc145f54e3aeb00f"
Jul 09 11:40:11.892757 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.892694    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f8fd72d5a709f514ccb50bddcff2d9828eb5a1f02d9da2dc145f54e3aeb00f"} err="failed to get container status \"c2f8fd72d5a709f514ccb50bddcff2d9828eb5a1f02d9da2dc145f54e3aeb00f\": rpc error: code = NotFound desc = could not find container \"c2f8fd72d5a709f514ccb50bddcff2d9828eb5a1f02d9da2dc145f54e3aeb00f\": container with ID starting with c2f8fd72d5a709f514ccb50bddcff2d9828eb5a1f02d9da2dc145f54e3aeb00f not found: ID does not exist"
Jul 09 11:40:11.897876 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.897854    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"]
Jul 09 11:40:11.903414 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.903395    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ca6bd9c3-openshift-default-796986c66f-npsdw"]
Jul 09 11:40:11.974686 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.974659    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"]
Jul 09 11:40:11.976606 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:40:11.976581    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6f0c42_1d7b_49b2_b915_c64f7450f962.slice/crio-5b36793a4484126ae4b8e68230b3770632b7778b9baacce8d4371a623d7ee334 WatchSource:0}: Error finding container 5b36793a4484126ae4b8e68230b3770632b7778b9baacce8d4371a623d7ee334: Status 404 returned error can't find the container with id 5b36793a4484126ae4b8e68230b3770632b7778b9baacce8d4371a623d7ee334
Jul 09 11:40:11.978718 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.978690    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:40:11.978795 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.978758    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:40:11.978832 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:11.978807    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:40:12.421425 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.421384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:12.614544 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.614510    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:12.737194 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.737105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:12.803977 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.803945    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"]
Jul 09 11:40:12.804254 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.804241    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" containerName="istio-proxy"
Jul 09 11:40:12.804315 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.804255    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" containerName="istio-proxy"
Jul 09 11:40:12.806865 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.806842    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"
Jul 09 11:40:12.809453 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.809430    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-73012e4a-serving-cert\""
Jul 09 11:40:12.824200 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.824175    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"]
Jul 09 11:40:12.832585 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.832559    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:12.844407 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.844385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cddm7\" (UniqueName: \"kubernetes.io/projected/125b45bd-744e-49b1-b656-5f915ab24aa7-kube-api-access-cddm7\") pod \"maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5\" (UID: \"125b45bd-744e-49b1-b656-5f915ab24aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"
Jul 09 11:40:12.844542 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.844439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125b45bd-744e-49b1-b656-5f915ab24aa7-maas-api-tls\") pod \"maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5\" (UID: \"125b45bd-744e-49b1-b656-5f915ab24aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"
Jul 09 11:40:12.844542 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.844530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125b45bd-744e-49b1-b656-5f915ab24aa7-openshift-service-ca\") pod \"maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5\" (UID: \"125b45bd-744e-49b1-b656-5f915ab24aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"
Jul 09 11:40:12.878629 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.878597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7" event={"ID":"2c6f0c42-1d7b-49b2-b915-c64f7450f962","Type":"ContainerStarted","Data":"6bd87c3aa0b94cf8ca3f80fae2739e91e1e02f380c95557375d4aad71eb28715"}
Jul 09 11:40:12.878809 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.878658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7" event={"ID":"2c6f0c42-1d7b-49b2-b915-c64f7450f962","Type":"ContainerStarted","Data":"5b36793a4484126ae4b8e68230b3770632b7778b9baacce8d4371a623d7ee334"}
Jul 09 11:40:12.910166 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.910112    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7" podStartSLOduration=1.9100956519999999 podStartE2EDuration="1.910095652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:40:12.908529425 +0000 UTC m=+1696.500260592" watchObservedRunningTime="2026-07-09 11:40:12.910095652 +0000 UTC m=+1696.501826816"
Jul 09 11:40:12.943357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.943325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:12.945562 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.945539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125b45bd-744e-49b1-b656-5f915ab24aa7-maas-api-tls\") pod \"maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5\" (UID: \"125b45bd-744e-49b1-b656-5f915ab24aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"
Jul 09 11:40:12.945636 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.945602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125b45bd-744e-49b1-b656-5f915ab24aa7-openshift-service-ca\") pod \"maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5\" (UID: \"125b45bd-744e-49b1-b656-5f915ab24aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"
Jul 09 11:40:12.945724 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.945709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cddm7\" (UniqueName: \"kubernetes.io/projected/125b45bd-744e-49b1-b656-5f915ab24aa7-kube-api-access-cddm7\") pod \"maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5\" (UID: \"125b45bd-744e-49b1-b656-5f915ab24aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"
Jul 09 11:40:12.945777 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:40:12.945731    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-serving-cert: secret "maas-api-e2e-derive-73012e4a-serving-cert" not found
Jul 09 11:40:12.945851 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:40:12.945828    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125b45bd-744e-49b1-b656-5f915ab24aa7-maas-api-tls podName:125b45bd-744e-49b1-b656-5f915ab24aa7 nodeName:}" failed. No retries permitted until 2026-07-09 11:40:13.44580417 +0000 UTC m=+1697.037535329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/125b45bd-744e-49b1-b656-5f915ab24aa7-maas-api-tls") pod "maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5" (UID: "125b45bd-744e-49b1-b656-5f915ab24aa7") : secret "maas-api-e2e-derive-73012e4a-serving-cert" not found
Jul 09 11:40:12.946405 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.946381    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125b45bd-744e-49b1-b656-5f915ab24aa7-openshift-service-ca\") pod \"maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5\" (UID: \"125b45bd-744e-49b1-b656-5f915ab24aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"
Jul 09 11:40:12.951267 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.951244    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4a218a8-ce05-4c4a-afe7-a50ae92a45cd" path="/var/lib/kubelet/pods/f4a218a8-ce05-4c4a-afe7-a50ae92a45cd/volumes"
Jul 09 11:40:12.957661 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:12.957638    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cddm7\" (UniqueName: \"kubernetes.io/projected/125b45bd-744e-49b1-b656-5f915ab24aa7-kube-api-access-cddm7\") pod \"maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5\" (UID: \"125b45bd-744e-49b1-b656-5f915ab24aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"
Jul 09 11:40:13.281538 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:13.281464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:13.450975 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:13.450938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125b45bd-744e-49b1-b656-5f915ab24aa7-maas-api-tls\") pod \"maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5\" (UID: \"125b45bd-744e-49b1-b656-5f915ab24aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"
Jul 09 11:40:13.453425 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:13.453403    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125b45bd-744e-49b1-b656-5f915ab24aa7-maas-api-tls\") pod \"maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5\" (UID: \"125b45bd-744e-49b1-b656-5f915ab24aa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"
Jul 09 11:40:13.718749 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:13.718654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"
Jul 09 11:40:13.727797 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:13.727747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:13.830617 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:13.830587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:13.835609 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:13.835584    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:13.840811 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:13.840766    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:13.853928 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:13.853909    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"]
Jul 09 11:40:13.856996 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:40:13.856950    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125b45bd_744e_49b1_b656_5f915ab24aa7.slice/crio-096bb5074138f4e5310ecf59148047bf91f3dcecc14c9bf6f1e1d281b2429563 WatchSource:0}: Error finding container 096bb5074138f4e5310ecf59148047bf91f3dcecc14c9bf6f1e1d281b2429563: Status 404 returned error can't find the container with id 096bb5074138f4e5310ecf59148047bf91f3dcecc14c9bf6f1e1d281b2429563
Jul 09 11:40:13.887476 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:13.887435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5" event={"ID":"125b45bd-744e-49b1-b656-5f915ab24aa7","Type":"ContainerStarted","Data":"096bb5074138f4e5310ecf59148047bf91f3dcecc14c9bf6f1e1d281b2429563"}
Jul 09 11:40:13.887801 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:13.887758    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:13.888867 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:13.888852    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:14.893675 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:14.893637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5" event={"ID":"125b45bd-744e-49b1-b656-5f915ab24aa7","Type":"ContainerStarted","Data":"ec33b8f8a47a441cbe16d47f8f0d41db042ed7927829fb52ce535fd32f95108b"}
Jul 09 11:40:14.913290 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:14.913242    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5" podStartSLOduration=2.324785441 podStartE2EDuration="2.913222961s" totalImagesPullingTime="588.43752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:40:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:40:14.911275207 +0000 UTC m=+1698.503006371" watchObservedRunningTime="2026-07-09 11:40:14.913222961 +0000 UTC m=+1698.504954126"
Jul 09 11:40:15.898206 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:15.898177    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"
Jul 09 11:40:16.672322 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:16.672294    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:17.076889 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:17.076853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:17.235622 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:17.235587    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"]
Jul 09 11:40:17.235839 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:17.235793    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7" podUID="2c6f0c42-1d7b-49b2-b915-c64f7450f962" containerName="istio-proxy" containerID="cri-o://6bd87c3aa0b94cf8ca3f80fae2739e91e1e02f380c95557375d4aad71eb28715" gracePeriod=30
Jul 09 11:40:17.262702 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:17.262673    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:17.357880 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:17.357776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:17.427264 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:17.427232    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"]
Jul 09 11:40:17.459017 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:17.458990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:17.907675 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:17.907626    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5" podUID="125b45bd-744e-49b1-b656-5f915ab24aa7" containerName="maas-api" containerID="cri-o://ec33b8f8a47a441cbe16d47f8f0d41db042ed7927829fb52ce535fd32f95108b" gracePeriod=30
Jul 09 11:40:18.147395 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.147365    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"
Jul 09 11:40:18.191615 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.191534    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125b45bd-744e-49b1-b656-5f915ab24aa7-openshift-service-ca\") pod \"125b45bd-744e-49b1-b656-5f915ab24aa7\" (UID: \"125b45bd-744e-49b1-b656-5f915ab24aa7\") "
Jul 09 11:40:18.191615 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.191607    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cddm7\" (UniqueName: \"kubernetes.io/projected/125b45bd-744e-49b1-b656-5f915ab24aa7-kube-api-access-cddm7\") pod \"125b45bd-744e-49b1-b656-5f915ab24aa7\" (UID: \"125b45bd-744e-49b1-b656-5f915ab24aa7\") "
Jul 09 11:40:18.191861 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.191629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125b45bd-744e-49b1-b656-5f915ab24aa7-maas-api-tls\") pod \"125b45bd-744e-49b1-b656-5f915ab24aa7\" (UID: \"125b45bd-744e-49b1-b656-5f915ab24aa7\") "
Jul 09 11:40:18.191929 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.191889    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125b45bd-744e-49b1-b656-5f915ab24aa7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "125b45bd-744e-49b1-b656-5f915ab24aa7" (UID: "125b45bd-744e-49b1-b656-5f915ab24aa7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:40:18.193912 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.193888    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125b45bd-744e-49b1-b656-5f915ab24aa7-kube-api-access-cddm7" (OuterVolumeSpecName: "kube-api-access-cddm7") pod "125b45bd-744e-49b1-b656-5f915ab24aa7" (UID: "125b45bd-744e-49b1-b656-5f915ab24aa7"). InnerVolumeSpecName "kube-api-access-cddm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:40:18.194035 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.193912    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125b45bd-744e-49b1-b656-5f915ab24aa7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "125b45bd-744e-49b1-b656-5f915ab24aa7" (UID: "125b45bd-744e-49b1-b656-5f915ab24aa7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 11:40:18.292849 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.292774    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cddm7\" (UniqueName: \"kubernetes.io/projected/125b45bd-744e-49b1-b656-5f915ab24aa7-kube-api-access-cddm7\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:18.292849 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.292843    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/125b45bd-744e-49b1-b656-5f915ab24aa7-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:18.292849 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.292857    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/125b45bd-744e-49b1-b656-5f915ab24aa7-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:18.912627 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.912594    2563 generic.go:356] "Generic (PLEG): container finished" podID="125b45bd-744e-49b1-b656-5f915ab24aa7" containerID="ec33b8f8a47a441cbe16d47f8f0d41db042ed7927829fb52ce535fd32f95108b" exitCode=0
Jul 09 11:40:18.912833 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.912645    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"
Jul 09 11:40:18.912833 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.912678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5" event={"ID":"125b45bd-744e-49b1-b656-5f915ab24aa7","Type":"ContainerDied","Data":"ec33b8f8a47a441cbe16d47f8f0d41db042ed7927829fb52ce535fd32f95108b"}
Jul 09 11:40:18.912833 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.912733    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5" event={"ID":"125b45bd-744e-49b1-b656-5f915ab24aa7","Type":"ContainerDied","Data":"096bb5074138f4e5310ecf59148047bf91f3dcecc14c9bf6f1e1d281b2429563"}
Jul 09 11:40:18.912833 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.912756    2563 scope.go:117] "RemoveContainer" containerID="ec33b8f8a47a441cbe16d47f8f0d41db042ed7927829fb52ce535fd32f95108b"
Jul 09 11:40:18.921766 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.921751    2563 scope.go:117] "RemoveContainer" containerID="ec33b8f8a47a441cbe16d47f8f0d41db042ed7927829fb52ce535fd32f95108b"
Jul 09 11:40:18.922026 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:40:18.922008    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec33b8f8a47a441cbe16d47f8f0d41db042ed7927829fb52ce535fd32f95108b\": container with ID starting with ec33b8f8a47a441cbe16d47f8f0d41db042ed7927829fb52ce535fd32f95108b not found: ID does not exist" containerID="ec33b8f8a47a441cbe16d47f8f0d41db042ed7927829fb52ce535fd32f95108b"
Jul 09 11:40:18.922074 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.922032    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec33b8f8a47a441cbe16d47f8f0d41db042ed7927829fb52ce535fd32f95108b"} err="failed to get container status \"ec33b8f8a47a441cbe16d47f8f0d41db042ed7927829fb52ce535fd32f95108b\": rpc error: code = NotFound desc = could not find container \"ec33b8f8a47a441cbe16d47f8f0d41db042ed7927829fb52ce535fd32f95108b\": container with ID starting with ec33b8f8a47a441cbe16d47f8f0d41db042ed7927829fb52ce535fd32f95108b not found: ID does not exist"
Jul 09 11:40:18.933834 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.933814    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"]
Jul 09 11:40:18.942278 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.942248    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-73012e4a-7486476c4f-cfnq5"]
Jul 09 11:40:18.951406 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:18.951380    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="125b45bd-744e-49b1-b656-5f915ab24aa7" path="/var/lib/kubelet/pods/125b45bd-744e-49b1-b656-5f915ab24aa7/volumes"
Jul 09 11:40:19.630322 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:19.630291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:22.495140 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.495116    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:22.629777 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.629736    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-envoy\") pod \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") "
Jul 09 11:40:22.629777 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.629775    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-credential-socket\") pod \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") "
Jul 09 11:40:22.630044 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.629849    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xn9fx\" (UniqueName: \"kubernetes.io/projected/2c6f0c42-1d7b-49b2-b915-c64f7450f962-kube-api-access-xn9fx\") pod \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") "
Jul 09 11:40:22.630044 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.629880    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-token\") pod \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") "
Jul 09 11:40:22.630044 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.629916    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-workload-certs\") pod \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") "
Jul 09 11:40:22.630044 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.629943    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-data\") pod \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") "
Jul 09 11:40:22.630044 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.629987    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istiod-ca-cert\") pod \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") "
Jul 09 11:40:22.630044 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.630018    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-podinfo\") pod \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") "
Jul 09 11:40:22.630335 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.630064    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-workload-socket\") pod \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\" (UID: \"2c6f0c42-1d7b-49b2-b915-c64f7450f962\") "
Jul 09 11:40:22.630335 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.630101    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2c6f0c42-1d7b-49b2-b915-c64f7450f962" (UID: "2c6f0c42-1d7b-49b2-b915-c64f7450f962"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:40:22.630335 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.630248    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-data" (OuterVolumeSpecName: "istio-data") pod "2c6f0c42-1d7b-49b2-b915-c64f7450f962" (UID: "2c6f0c42-1d7b-49b2-b915-c64f7450f962"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:40:22.630335 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.630270    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2c6f0c42-1d7b-49b2-b915-c64f7450f962" (UID: "2c6f0c42-1d7b-49b2-b915-c64f7450f962"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:40:22.630512 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.630365    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:22.630512 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.630372    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2c6f0c42-1d7b-49b2-b915-c64f7450f962" (UID: "2c6f0c42-1d7b-49b2-b915-c64f7450f962"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:40:22.630512 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.630385    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:22.630512 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.630399    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:22.630512 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.630475    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2c6f0c42-1d7b-49b2-b915-c64f7450f962" (UID: "2c6f0c42-1d7b-49b2-b915-c64f7450f962"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:40:22.632123 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.632090    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6f0c42-1d7b-49b2-b915-c64f7450f962-kube-api-access-xn9fx" (OuterVolumeSpecName: "kube-api-access-xn9fx") pod "2c6f0c42-1d7b-49b2-b915-c64f7450f962" (UID: "2c6f0c42-1d7b-49b2-b915-c64f7450f962"). InnerVolumeSpecName "kube-api-access-xn9fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:40:22.632237 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.632158    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2c6f0c42-1d7b-49b2-b915-c64f7450f962" (UID: "2c6f0c42-1d7b-49b2-b915-c64f7450f962"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:40:22.632476 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.632454    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2c6f0c42-1d7b-49b2-b915-c64f7450f962" (UID: "2c6f0c42-1d7b-49b2-b915-c64f7450f962"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 11:40:22.632476 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.632462    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-token" (OuterVolumeSpecName: "istio-token") pod "2c6f0c42-1d7b-49b2-b915-c64f7450f962" (UID: "2c6f0c42-1d7b-49b2-b915-c64f7450f962"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:40:22.731215 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.731179    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:22.731215 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.731210    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:22.731215 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.731220    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:22.731455 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.731229    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:22.731455 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.731238    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn9fx\" (UniqueName: \"kubernetes.io/projected/2c6f0c42-1d7b-49b2-b915-c64f7450f962-kube-api-access-xn9fx\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:22.731455 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.731246    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c6f0c42-1d7b-49b2-b915-c64f7450f962-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:40:22.931486 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.931389    2563 generic.go:356] "Generic (PLEG): container finished" podID="2c6f0c42-1d7b-49b2-b915-c64f7450f962" containerID="6bd87c3aa0b94cf8ca3f80fae2739e91e1e02f380c95557375d4aad71eb28715" exitCode=0
Jul 09 11:40:22.931486 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.931434    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"
Jul 09 11:40:22.931719 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.931483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7" event={"ID":"2c6f0c42-1d7b-49b2-b915-c64f7450f962","Type":"ContainerDied","Data":"6bd87c3aa0b94cf8ca3f80fae2739e91e1e02f380c95557375d4aad71eb28715"}
Jul 09 11:40:22.931719 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.931547    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7" event={"ID":"2c6f0c42-1d7b-49b2-b915-c64f7450f962","Type":"ContainerDied","Data":"5b36793a4484126ae4b8e68230b3770632b7778b9baacce8d4371a623d7ee334"}
Jul 09 11:40:22.931719 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.931575    2563 scope.go:117] "RemoveContainer" containerID="6bd87c3aa0b94cf8ca3f80fae2739e91e1e02f380c95557375d4aad71eb28715"
Jul 09 11:40:22.940516 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.940499    2563 scope.go:117] "RemoveContainer" containerID="6bd87c3aa0b94cf8ca3f80fae2739e91e1e02f380c95557375d4aad71eb28715"
Jul 09 11:40:22.940777 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:40:22.940760    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd87c3aa0b94cf8ca3f80fae2739e91e1e02f380c95557375d4aad71eb28715\": container with ID starting with 6bd87c3aa0b94cf8ca3f80fae2739e91e1e02f380c95557375d4aad71eb28715 not found: ID does not exist" containerID="6bd87c3aa0b94cf8ca3f80fae2739e91e1e02f380c95557375d4aad71eb28715"
Jul 09 11:40:22.940851 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.940802    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd87c3aa0b94cf8ca3f80fae2739e91e1e02f380c95557375d4aad71eb28715"} err="failed to get container status \"6bd87c3aa0b94cf8ca3f80fae2739e91e1e02f380c95557375d4aad71eb28715\": rpc error: code = NotFound desc = could not find container \"6bd87c3aa0b94cf8ca3f80fae2739e91e1e02f380c95557375d4aad71eb28715\": container with ID starting with 6bd87c3aa0b94cf8ca3f80fae2739e91e1e02f380c95557375d4aad71eb28715 not found: ID does not exist"
Jul 09 11:40:22.953177 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.953153    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"]
Jul 09 11:40:22.955714 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:22.955693    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-73012e4a-openshift-default-59b6ff47b7-l8lj7"]
Jul 09 11:40:23.333842 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:23.333807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:23.397706 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:23.397672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:23.787927 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:23.787900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:23.887029 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:23.886984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:23.988299 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:23.988263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:24.949847 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:24.949808    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c6f0c42-1d7b-49b2-b915-c64f7450f962" path="/var/lib/kubelet/pods/2c6f0c42-1d7b-49b2-b915-c64f7450f962/volumes"
Jul 09 11:40:25.400187 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:25.400156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:31.443268 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:31.443222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:31.549583 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:31.549546    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:31.644619 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:31.644574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:31.746549 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:31.746451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:32.149885 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:32.149823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:40:58.105405 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:40:58.105372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:04.345719 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:04.345683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:04.455973 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:04.455935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:04.559916 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:04.559883    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:19.469801 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:19.469755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:19.571312 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:19.571278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:20.511623 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:20.511580    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:26.713831 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:26.713771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:26.816733 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:26.816698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:26.911711 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:26.911675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:37.465792 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:37.465742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:38.349336 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:38.349299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:44.314269 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:44.314238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:44.394989 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:44.394957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:44.506096 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:44.506060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:44.597184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:44.597096    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:45.458370 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:45.458340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:45.559920 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:45.559887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:47.247697 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:47.247621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:53.314345 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:53.314309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:53.390291 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:53.390259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:53.992584 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:53.992550    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:41:56.923905 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:56.923871    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:41:56.927803 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:41:56.927756    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:42:00.214730 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.214698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:00.296545 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.296511    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:00.475598 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.475518    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:00.745504 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.745426    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"]
Jul 09 11:42:00.745759 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.745745    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c6f0c42-1d7b-49b2-b915-c64f7450f962" containerName="istio-proxy"
Jul 09 11:42:00.745759 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.745759    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c6f0c42-1d7b-49b2-b915-c64f7450f962" containerName="istio-proxy"
Jul 09 11:42:00.745878 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.745767    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="125b45bd-744e-49b1-b656-5f915ab24aa7" containerName="maas-api"
Jul 09 11:42:00.745878 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.745773    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="125b45bd-744e-49b1-b656-5f915ab24aa7" containerName="maas-api"
Jul 09 11:42:00.745878 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.745852    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c6f0c42-1d7b-49b2-b915-c64f7450f962" containerName="istio-proxy"
Jul 09 11:42:00.745878 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.745862    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="125b45bd-744e-49b1-b656-5f915ab24aa7" containerName="maas-api"
Jul 09 11:42:00.747715 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.747700    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:00.750458 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.750430    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3b82f7ea-openshift-default-dockercfg-xp992\""
Jul 09 11:42:00.771848 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.771817    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"]
Jul 09 11:42:00.775733 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.775695    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:00.914207 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.914168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-workload-certs\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:00.914207 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.914211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-workload-socket\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:00.914447 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.914238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-envoy\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:00.914447 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.914313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fq5c\" (UniqueName: \"kubernetes.io/projected/4971576f-49f7-4400-bc8f-f52de0a6b9e1-kube-api-access-6fq5c\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:00.914447 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.914366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-credential-socket\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:00.914447 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.914390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-token\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:00.914447 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.914419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-podinfo\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:00.914447 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.914433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istiod-ca-cert\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:00.914635 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:00.914459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-data\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.015311 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.015271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-credential-socket\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.015311 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.015322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-token\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.015597 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.015346    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-podinfo\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.015597 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.015362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istiod-ca-cert\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.015597 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.015380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-data\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.015597 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.015437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-workload-certs\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.015597 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.015457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-workload-socket\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.015597 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.015498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-envoy\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.015597 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.015532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fq5c\" (UniqueName: \"kubernetes.io/projected/4971576f-49f7-4400-bc8f-f52de0a6b9e1-kube-api-access-6fq5c\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.015976 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.015812    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-credential-socket\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.016133 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.016107    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-data\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.016300 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.016136    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-workload-certs\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.016300 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.016171    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istiod-ca-cert\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.016428 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.016197    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-workload-socket\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.018077 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.018052    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-envoy\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.018183 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.018164    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-podinfo\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.030439 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.030414    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-token\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.032821 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.032794    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fq5c\" (UniqueName: \"kubernetes.io/projected/4971576f-49f7-4400-bc8f-f52de0a6b9e1-kube-api-access-6fq5c\") pod \"e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") " pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.057859 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.057822    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:01.207987 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.207774    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"]
Jul 09 11:42:01.211097 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:42:01.211059    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4971576f_49f7_4400_bc8f_f52de0a6b9e1.slice/crio-848e2051aa45349fd775d249632f48d91a2dbe0bfa5b15f11c1042cf69b6c16b WatchSource:0}: Error finding container 848e2051aa45349fd775d249632f48d91a2dbe0bfa5b15f11c1042cf69b6c16b: Status 404 returned error can't find the container with id 848e2051aa45349fd775d249632f48d91a2dbe0bfa5b15f11c1042cf69b6c16b
Jul 09 11:42:01.213191 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.213161    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:42:01.213284 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.213228    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:42:01.213284 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.213256    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:42:01.325604 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.325522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:01.364296 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.364260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w" event={"ID":"4971576f-49f7-4400-bc8f-f52de0a6b9e1","Type":"ContainerStarted","Data":"0ae879ebb1f3f3f299be2f9eefe2a66c39b8902572c0e5ac9c0eea7b59eee432"}
Jul 09 11:42:01.364504 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.364320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w" event={"ID":"4971576f-49f7-4400-bc8f-f52de0a6b9e1","Type":"ContainerStarted","Data":"848e2051aa45349fd775d249632f48d91a2dbe0bfa5b15f11c1042cf69b6c16b"}
Jul 09 11:42:01.386698 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:01.386561    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w" podStartSLOduration=1.386543234 podStartE2EDuration="1.386543234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:42:01.384227895 +0000 UTC m=+1804.975959073" watchObservedRunningTime="2026-07-09 11:42:01.386543234 +0000 UTC m=+1804.978274461"
Jul 09 11:42:02.058055 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:02.058019    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:02.063052 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:02.063026    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:02.369871 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:02.369761    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:02.370769 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:02.370750    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:03.563970 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:03.563927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:03.667951 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:03.667912    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:03.809524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:03.809488    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"]
Jul 09 11:42:03.816254 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:03.816159    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"
Jul 09 11:42:03.819573 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:03.819548    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3b82f7ea-serving-cert\""
Jul 09 11:42:03.827406 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:03.827377    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"]
Jul 09 11:42:03.848773 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:03.848746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:03.929924 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:03.929889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:03.939368 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:03.939338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps8tt\" (UniqueName: \"kubernetes.io/projected/e261f308-f344-4446-a258-04874ba03fe0-kube-api-access-ps8tt\") pod \"maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk\" (UID: \"e261f308-f344-4446-a258-04874ba03fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"
Jul 09 11:42:03.939539 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:03.939384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e261f308-f344-4446-a258-04874ba03fe0-openshift-service-ca\") pod \"maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk\" (UID: \"e261f308-f344-4446-a258-04874ba03fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"
Jul 09 11:42:03.939539 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:03.939488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e261f308-f344-4446-a258-04874ba03fe0-maas-api-tls\") pod \"maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk\" (UID: \"e261f308-f344-4446-a258-04874ba03fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"
Jul 09 11:42:04.040987 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:04.040952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e261f308-f344-4446-a258-04874ba03fe0-maas-api-tls\") pod \"maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk\" (UID: \"e261f308-f344-4446-a258-04874ba03fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"
Jul 09 11:42:04.041182 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:04.041026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps8tt\" (UniqueName: \"kubernetes.io/projected/e261f308-f344-4446-a258-04874ba03fe0-kube-api-access-ps8tt\") pod \"maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk\" (UID: \"e261f308-f344-4446-a258-04874ba03fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"
Jul 09 11:42:04.041272 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:04.041181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e261f308-f344-4446-a258-04874ba03fe0-openshift-service-ca\") pod \"maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk\" (UID: \"e261f308-f344-4446-a258-04874ba03fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"
Jul 09 11:42:04.041949 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:04.041922    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e261f308-f344-4446-a258-04874ba03fe0-openshift-service-ca\") pod \"maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk\" (UID: \"e261f308-f344-4446-a258-04874ba03fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"
Jul 09 11:42:04.043567 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:04.043537    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e261f308-f344-4446-a258-04874ba03fe0-maas-api-tls\") pod \"maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk\" (UID: \"e261f308-f344-4446-a258-04874ba03fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"
Jul 09 11:42:04.050305 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:04.050278    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps8tt\" (UniqueName: \"kubernetes.io/projected/e261f308-f344-4446-a258-04874ba03fe0-kube-api-access-ps8tt\") pod \"maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk\" (UID: \"e261f308-f344-4446-a258-04874ba03fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"
Jul 09 11:42:04.128987 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:04.128900    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"
Jul 09 11:42:04.279674 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:04.279645    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"]
Jul 09 11:42:04.281578 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:42:04.281549    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode261f308_f344_4446_a258_04874ba03fe0.slice/crio-2a9748bdf3e5164133a82f83358919b7d4c6a13ad005e719bbf5bb43afe9a11b WatchSource:0}: Error finding container 2a9748bdf3e5164133a82f83358919b7d4c6a13ad005e719bbf5bb43afe9a11b: Status 404 returned error can't find the container with id 2a9748bdf3e5164133a82f83358919b7d4c6a13ad005e719bbf5bb43afe9a11b
Jul 09 11:42:04.379910 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:04.379817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk" event={"ID":"e261f308-f344-4446-a258-04874ba03fe0","Type":"ContainerStarted","Data":"2a9748bdf3e5164133a82f83358919b7d4c6a13ad005e719bbf5bb43afe9a11b"}
Jul 09 11:42:04.483771 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:04.483737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:04.583985 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:04.583952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:05.385726 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:05.385688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk" event={"ID":"e261f308-f344-4446-a258-04874ba03fe0","Type":"ContainerStarted","Data":"46e0e71e4a640732b188f15a169a4a580bc014fe2723632934ceffac6ea71d1f"}
Jul 09 11:42:05.385935 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:05.385824    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"
Jul 09 11:42:05.406223 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:05.406168    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk" podStartSLOduration=1.977058064 podStartE2EDuration="2.406147393s" totalImagesPullingTime="429.089329ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:42:05.405553319 +0000 UTC m=+1808.997284484" watchObservedRunningTime="2026-07-09 11:42:05.406147393 +0000 UTC m=+1808.997878565"
Jul 09 11:42:08.057379 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:08.057347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:08.704109 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:08.704067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:08.876079 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:08.876050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:09.151850 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:09.151819    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:11.396490 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:11.396463    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"
Jul 09 11:42:15.366373 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:15.366340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:15.467264 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:15.467224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:15.809403 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:15.809376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:16.227972 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.227880    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"]
Jul 09 11:42:16.228224 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.228186    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk" podUID="e261f308-f344-4446-a258-04874ba03fe0" containerName="maas-api" containerID="cri-o://46e0e71e4a640732b188f15a169a4a580bc014fe2723632934ceffac6ea71d1f" gracePeriod=30
Jul 09 11:42:16.250860 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.250828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:16.316365 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.316336    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:16.391188 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.391149    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk" podUID="e261f308-f344-4446-a258-04874ba03fe0" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.34:8443/health\": dial tcp 10.134.0.34:8443: connect: connection refused"
Jul 09 11:42:16.435171 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.435134    2563 generic.go:356] "Generic (PLEG): container finished" podID="e261f308-f344-4446-a258-04874ba03fe0" containerID="46e0e71e4a640732b188f15a169a4a580bc014fe2723632934ceffac6ea71d1f" exitCode=0
Jul 09 11:42:16.436260 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.435205    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk" event={"ID":"e261f308-f344-4446-a258-04874ba03fe0","Type":"ContainerDied","Data":"46e0e71e4a640732b188f15a169a4a580bc014fe2723632934ceffac6ea71d1f"}
Jul 09 11:42:16.469570 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.469547    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"
Jul 09 11:42:16.541763 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.541738    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e261f308-f344-4446-a258-04874ba03fe0-maas-api-tls\") pod \"e261f308-f344-4446-a258-04874ba03fe0\" (UID: \"e261f308-f344-4446-a258-04874ba03fe0\") "
Jul 09 11:42:16.541928 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.541915    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps8tt\" (UniqueName: \"kubernetes.io/projected/e261f308-f344-4446-a258-04874ba03fe0-kube-api-access-ps8tt\") pod \"e261f308-f344-4446-a258-04874ba03fe0\" (UID: \"e261f308-f344-4446-a258-04874ba03fe0\") "
Jul 09 11:42:16.541972 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.541947    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e261f308-f344-4446-a258-04874ba03fe0-openshift-service-ca\") pod \"e261f308-f344-4446-a258-04874ba03fe0\" (UID: \"e261f308-f344-4446-a258-04874ba03fe0\") "
Jul 09 11:42:16.542302 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.542274    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e261f308-f344-4446-a258-04874ba03fe0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e261f308-f344-4446-a258-04874ba03fe0" (UID: "e261f308-f344-4446-a258-04874ba03fe0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:42:16.543602 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.543581    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e261f308-f344-4446-a258-04874ba03fe0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e261f308-f344-4446-a258-04874ba03fe0" (UID: "e261f308-f344-4446-a258-04874ba03fe0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 11:42:16.543812 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.543795    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e261f308-f344-4446-a258-04874ba03fe0-kube-api-access-ps8tt" (OuterVolumeSpecName: "kube-api-access-ps8tt") pod "e261f308-f344-4446-a258-04874ba03fe0" (UID: "e261f308-f344-4446-a258-04874ba03fe0"). InnerVolumeSpecName "kube-api-access-ps8tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:42:16.643377 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.643337    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps8tt\" (UniqueName: \"kubernetes.io/projected/e261f308-f344-4446-a258-04874ba03fe0-kube-api-access-ps8tt\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:42:16.643377 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.643374    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e261f308-f344-4446-a258-04874ba03fe0-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:42:16.643569 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:16.643388    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e261f308-f344-4446-a258-04874ba03fe0-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:42:17.371481 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:17.371445    2563 patch_prober.go:28] interesting pod/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 09 11:42:17.371655 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:17.371502    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w" podUID="4971576f-49f7-4400-bc8f-f52de0a6b9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 09 11:42:17.441078 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:17.441046    2563 generic.go:356] "Generic (PLEG): container finished" podID="4971576f-49f7-4400-bc8f-f52de0a6b9e1" containerID="0ae879ebb1f3f3f299be2f9eefe2a66c39b8902572c0e5ac9c0eea7b59eee432" exitCode=0
Jul 09 11:42:17.441463 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:17.441122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w" event={"ID":"4971576f-49f7-4400-bc8f-f52de0a6b9e1","Type":"ContainerDied","Data":"0ae879ebb1f3f3f299be2f9eefe2a66c39b8902572c0e5ac9c0eea7b59eee432"}
Jul 09 11:42:17.441542 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:17.441528    2563 scope.go:117] "RemoveContainer" containerID="0ae879ebb1f3f3f299be2f9eefe2a66c39b8902572c0e5ac9c0eea7b59eee432"
Jul 09 11:42:17.442490 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:17.442456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk" event={"ID":"e261f308-f344-4446-a258-04874ba03fe0","Type":"ContainerDied","Data":"2a9748bdf3e5164133a82f83358919b7d4c6a13ad005e719bbf5bb43afe9a11b"}
Jul 09 11:42:17.442490 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:17.442482    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"
Jul 09 11:42:17.442697 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:17.442509    2563 scope.go:117] "RemoveContainer" containerID="46e0e71e4a640732b188f15a169a4a580bc014fe2723632934ceffac6ea71d1f"
Jul 09 11:42:17.442697 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:17.442609    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:42:17.442697 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:17.442679    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:42:17.442888 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:17.442713    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:42:17.478908 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:17.478844    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"]
Jul 09 11:42:17.481471 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:17.481449    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b82f7ea-8669474f5b-7c9bk"]
Jul 09 11:42:18.452739 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:18.452687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w" event={"ID":"4971576f-49f7-4400-bc8f-f52de0a6b9e1","Type":"ContainerStarted","Data":"d54993050dfd874cb272be4b5ee11c073317731f01d196ba55fd9c388c5e3d34"}
Jul 09 11:42:18.955677 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:18.955627    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e261f308-f344-4446-a258-04874ba03fe0" path="/var/lib/kubelet/pods/e261f308-f344-4446-a258-04874ba03fe0/volumes"
Jul 09 11:42:19.058213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:19.058117    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:19.058851 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:19.058817    2563 patch_prober.go:28] interesting pod/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 09 11:42:19.058978 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:19.058886    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w" podUID="4971576f-49f7-4400-bc8f-f52de0a6b9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 09 11:42:20.058967 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:20.058486    2563 patch_prober.go:28] interesting pod/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 09 11:42:20.058967 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:20.058551    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w" podUID="4971576f-49f7-4400-bc8f-f52de0a6b9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 09 11:42:21.058808 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:21.058746    2563 patch_prober.go:28] interesting pod/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 09 11:42:21.059114 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:21.059085    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w" podUID="4971576f-49f7-4400-bc8f-f52de0a6b9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 09 11:42:22.058360 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:22.058319    2563 patch_prober.go:28] interesting pod/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 09 11:42:22.058669 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:22.058633    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w" podUID="4971576f-49f7-4400-bc8f-f52de0a6b9e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 09 11:42:22.388661 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:22.388570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:22.490696 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:22.490652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:22.649897 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:22.649812    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"]
Jul 09 11:42:22.650058 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:22.650002    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w" podUID="4971576f-49f7-4400-bc8f-f52de0a6b9e1" containerName="istio-proxy" containerID="cri-o://d54993050dfd874cb272be4b5ee11c073317731f01d196ba55fd9c388c5e3d34" gracePeriod=30
Jul 09 11:42:22.668821 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:22.668792    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:23.000136 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:23.000112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:23.105763 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:23.105735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:23.212661 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:23.212632    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:27.891333 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.891309    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:27.938753 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.938676    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-credential-socket\") pod \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") "
Jul 09 11:42:27.938753 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.938718    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istiod-ca-cert\") pod \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") "
Jul 09 11:42:27.938753 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.938736    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-data\") pod \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") "
Jul 09 11:42:27.939041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.938772    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-envoy\") pod \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") "
Jul 09 11:42:27.939041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.938832    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-token\") pod \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") "
Jul 09 11:42:27.939041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.938856    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-podinfo\") pod \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") "
Jul 09 11:42:27.939041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.938891    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-workload-certs\") pod \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") "
Jul 09 11:42:27.939041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.938941    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-workload-socket\") pod \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") "
Jul 09 11:42:27.939041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.938987    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fq5c\" (UniqueName: \"kubernetes.io/projected/4971576f-49f7-4400-bc8f-f52de0a6b9e1-kube-api-access-6fq5c\") pod \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\" (UID: \"4971576f-49f7-4400-bc8f-f52de0a6b9e1\") "
Jul 09 11:42:27.939359 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.939045    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4971576f-49f7-4400-bc8f-f52de0a6b9e1" (UID: "4971576f-49f7-4400-bc8f-f52de0a6b9e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:42:27.939359 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.939248    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4971576f-49f7-4400-bc8f-f52de0a6b9e1" (UID: "4971576f-49f7-4400-bc8f-f52de0a6b9e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:42:27.939359 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.939177    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4971576f-49f7-4400-bc8f-f52de0a6b9e1" (UID: "4971576f-49f7-4400-bc8f-f52de0a6b9e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:42:27.939359 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.939270    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:42:27.939561 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.939416    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4971576f-49f7-4400-bc8f-f52de0a6b9e1" (UID: "4971576f-49f7-4400-bc8f-f52de0a6b9e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:42:27.939747 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.939703    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-data" (OuterVolumeSpecName: "istio-data") pod "4971576f-49f7-4400-bc8f-f52de0a6b9e1" (UID: "4971576f-49f7-4400-bc8f-f52de0a6b9e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:42:27.941136 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.941104    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4971576f-49f7-4400-bc8f-f52de0a6b9e1" (UID: "4971576f-49f7-4400-bc8f-f52de0a6b9e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 11:42:27.941244 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.941225    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4971576f-49f7-4400-bc8f-f52de0a6b9e1-kube-api-access-6fq5c" (OuterVolumeSpecName: "kube-api-access-6fq5c") pod "4971576f-49f7-4400-bc8f-f52de0a6b9e1" (UID: "4971576f-49f7-4400-bc8f-f52de0a6b9e1"). InnerVolumeSpecName "kube-api-access-6fq5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:42:27.941328 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.941305    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-token" (OuterVolumeSpecName: "istio-token") pod "4971576f-49f7-4400-bc8f-f52de0a6b9e1" (UID: "4971576f-49f7-4400-bc8f-f52de0a6b9e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:42:27.941513 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:27.941496    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4971576f-49f7-4400-bc8f-f52de0a6b9e1" (UID: "4971576f-49f7-4400-bc8f-f52de0a6b9e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:42:28.039731 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.039692    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fq5c\" (UniqueName: \"kubernetes.io/projected/4971576f-49f7-4400-bc8f-f52de0a6b9e1-kube-api-access-6fq5c\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:42:28.039731 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.039732    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:42:28.040034 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.039746    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:42:28.040034 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.039758    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:42:28.040034 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.039771    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:42:28.040034 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.039807    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4971576f-49f7-4400-bc8f-f52de0a6b9e1-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:42:28.040034 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.039821    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:42:28.040034 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.039834    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4971576f-49f7-4400-bc8f-f52de0a6b9e1-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:42:28.502946 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.502911    2563 generic.go:356] "Generic (PLEG): container finished" podID="4971576f-49f7-4400-bc8f-f52de0a6b9e1" containerID="d54993050dfd874cb272be4b5ee11c073317731f01d196ba55fd9c388c5e3d34" exitCode=0
Jul 09 11:42:28.503135 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.502955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w" event={"ID":"4971576f-49f7-4400-bc8f-f52de0a6b9e1","Type":"ContainerDied","Data":"d54993050dfd874cb272be4b5ee11c073317731f01d196ba55fd9c388c5e3d34"}
Jul 09 11:42:28.503135 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.502963    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"
Jul 09 11:42:28.503135 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.502997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w" event={"ID":"4971576f-49f7-4400-bc8f-f52de0a6b9e1","Type":"ContainerDied","Data":"848e2051aa45349fd775d249632f48d91a2dbe0bfa5b15f11c1042cf69b6c16b"}
Jul 09 11:42:28.503135 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.503019    2563 scope.go:117] "RemoveContainer" containerID="d54993050dfd874cb272be4b5ee11c073317731f01d196ba55fd9c388c5e3d34"
Jul 09 11:42:28.511745 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.511728    2563 scope.go:117] "RemoveContainer" containerID="0ae879ebb1f3f3f299be2f9eefe2a66c39b8902572c0e5ac9c0eea7b59eee432"
Jul 09 11:42:28.519133 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.519114    2563 scope.go:117] "RemoveContainer" containerID="d54993050dfd874cb272be4b5ee11c073317731f01d196ba55fd9c388c5e3d34"
Jul 09 11:42:28.519397 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:42:28.519376    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d54993050dfd874cb272be4b5ee11c073317731f01d196ba55fd9c388c5e3d34\": container with ID starting with d54993050dfd874cb272be4b5ee11c073317731f01d196ba55fd9c388c5e3d34 not found: ID does not exist" containerID="d54993050dfd874cb272be4b5ee11c073317731f01d196ba55fd9c388c5e3d34"
Jul 09 11:42:28.519478 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.519409    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d54993050dfd874cb272be4b5ee11c073317731f01d196ba55fd9c388c5e3d34"} err="failed to get container status \"d54993050dfd874cb272be4b5ee11c073317731f01d196ba55fd9c388c5e3d34\": rpc error: code = NotFound desc = could not find container \"d54993050dfd874cb272be4b5ee11c073317731f01d196ba55fd9c388c5e3d34\": container with ID starting with d54993050dfd874cb272be4b5ee11c073317731f01d196ba55fd9c388c5e3d34 not found: ID does not exist"
Jul 09 11:42:28.519478 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.519435    2563 scope.go:117] "RemoveContainer" containerID="0ae879ebb1f3f3f299be2f9eefe2a66c39b8902572c0e5ac9c0eea7b59eee432"
Jul 09 11:42:28.519659 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:42:28.519642    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae879ebb1f3f3f299be2f9eefe2a66c39b8902572c0e5ac9c0eea7b59eee432\": container with ID starting with 0ae879ebb1f3f3f299be2f9eefe2a66c39b8902572c0e5ac9c0eea7b59eee432 not found: ID does not exist" containerID="0ae879ebb1f3f3f299be2f9eefe2a66c39b8902572c0e5ac9c0eea7b59eee432"
Jul 09 11:42:28.519701 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.519664    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae879ebb1f3f3f299be2f9eefe2a66c39b8902572c0e5ac9c0eea7b59eee432"} err="failed to get container status \"0ae879ebb1f3f3f299be2f9eefe2a66c39b8902572c0e5ac9c0eea7b59eee432\": rpc error: code = NotFound desc = could not find container \"0ae879ebb1f3f3f299be2f9eefe2a66c39b8902572c0e5ac9c0eea7b59eee432\": container with ID starting with 0ae879ebb1f3f3f299be2f9eefe2a66c39b8902572c0e5ac9c0eea7b59eee432 not found: ID does not exist"
Jul 09 11:42:28.527852 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.527831    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"]
Jul 09 11:42:28.531806 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.531771    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3b82f7ea-openshift-default-59fbdcc587-rx95w"]
Jul 09 11:42:28.949673 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:28.949588    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4971576f-49f7-4400-bc8f-f52de0a6b9e1" path="/var/lib/kubelet/pods/4971576f-49f7-4400-bc8f-f52de0a6b9e1/volumes"
Jul 09 11:42:29.708342 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:29.708306    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:29.810368 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:29.810332    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:29.956567 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:29.956536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:30.081443 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:30.081406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:36.352901 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:36.352866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:36.462240 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:36.462204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:43.279339 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.279304    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:43.578316 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.578236    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"]
Jul 09 11:42:43.578599 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.578583    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4971576f-49f7-4400-bc8f-f52de0a6b9e1" containerName="istio-proxy"
Jul 09 11:42:43.578599 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.578600    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4971576f-49f7-4400-bc8f-f52de0a6b9e1" containerName="istio-proxy"
Jul 09 11:42:43.578730 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.578616    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4971576f-49f7-4400-bc8f-f52de0a6b9e1" containerName="istio-proxy"
Jul 09 11:42:43.578730 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.578622    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4971576f-49f7-4400-bc8f-f52de0a6b9e1" containerName="istio-proxy"
Jul 09 11:42:43.578730 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.578634    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e261f308-f344-4446-a258-04874ba03fe0" containerName="maas-api"
Jul 09 11:42:43.578730 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.578641    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e261f308-f344-4446-a258-04874ba03fe0" containerName="maas-api"
Jul 09 11:42:43.578730 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.578689    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4971576f-49f7-4400-bc8f-f52de0a6b9e1" containerName="istio-proxy"
Jul 09 11:42:43.578730 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.578695    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e261f308-f344-4446-a258-04874ba03fe0" containerName="maas-api"
Jul 09 11:42:43.583231 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.583208    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.586058 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.586040    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b5eec9ef-openshift-default-dockercfg-n5qnh\""
Jul 09 11:42:43.596652 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.596626    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"]
Jul 09 11:42:43.619754 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.619728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:43.676357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.676326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61104450-8f60-4486-9f29-717f3148a5fc-istio-token\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.676568 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.676377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-workload-certs\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.676568 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.676407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-istio-data\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.676568 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.676440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-workload-socket\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.676568 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.676468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61104450-8f60-4486-9f29-717f3148a5fc-istio-podinfo\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.676568 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.676489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61104450-8f60-4486-9f29-717f3148a5fc-istiod-ca-cert\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.676568 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.676509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-istio-envoy\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.676568 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.676550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vf4m\" (UniqueName: \"kubernetes.io/projected/61104450-8f60-4486-9f29-717f3148a5fc-kube-api-access-6vf4m\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.676966 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.676615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-credential-socket\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.777953 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.777911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vf4m\" (UniqueName: \"kubernetes.io/projected/61104450-8f60-4486-9f29-717f3148a5fc-kube-api-access-6vf4m\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.777953 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.777954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-credential-socket\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.778213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.777979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61104450-8f60-4486-9f29-717f3148a5fc-istio-token\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.778213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.778028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-workload-certs\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.778213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.778065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-istio-data\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.778213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.778101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-workload-socket\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.778213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.778133    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61104450-8f60-4486-9f29-717f3148a5fc-istio-podinfo\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.778213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.778162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61104450-8f60-4486-9f29-717f3148a5fc-istiod-ca-cert\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.778213 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.778192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-istio-envoy\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.778569 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.778434    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-workload-certs\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.778569 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.778473    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-workload-socket\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.778569 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.778529    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-istio-data\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.778569 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.778559    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-credential-socket\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.779043 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.779018    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61104450-8f60-4486-9f29-717f3148a5fc-istiod-ca-cert\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.780568 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.780543    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-istio-envoy\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.780815 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.780775    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61104450-8f60-4486-9f29-717f3148a5fc-istio-podinfo\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.786453 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.786430    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61104450-8f60-4486-9f29-717f3148a5fc-istio-token\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.786979 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.786958    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vf4m\" (UniqueName: \"kubernetes.io/projected/61104450-8f60-4486-9f29-717f3148a5fc-kube-api-access-6vf4m\") pod \"e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") " pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:43.862834 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.862738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:43.893966 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:43.893930    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:44.029044 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:44.029017    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"]
Jul 09 11:42:44.034052 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:44.033997    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:42:44.034214 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:44.034142    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:42:44.034295 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:44.034256    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:42:44.574378 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:44.574341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs" event={"ID":"61104450-8f60-4486-9f29-717f3148a5fc","Type":"ContainerStarted","Data":"e1ec7c4299d7a55def1915515400844a27d7c48cc0539a4a8dbcb1e70e98a034"}
Jul 09 11:42:44.574905 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:44.574441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs" event={"ID":"61104450-8f60-4486-9f29-717f3148a5fc","Type":"ContainerStarted","Data":"6be4ff5d0c161a625acd57c2ef92f498e70a672cb6743dcb286cc74209b0a0a1"}
Jul 09 11:42:44.606535 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:44.606480    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs" podStartSLOduration=1.606464501 podStartE2EDuration="1.606464501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:42:44.60534722 +0000 UTC m=+1848.197078388" watchObservedRunningTime="2026-07-09 11:42:44.606464501 +0000 UTC m=+1848.198195664"
Jul 09 11:42:44.894259 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:44.894178    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:45.899883 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:45.899852    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:46.582998 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.582953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:46.583902 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.583886    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:46.584794 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.584762    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:42:46.639709 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.639681    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"]
Jul 09 11:42:46.640083 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.640068    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4971576f-49f7-4400-bc8f-f52de0a6b9e1" containerName="istio-proxy"
Jul 09 11:42:46.643090 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.643069    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:42:46.645981 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.645958    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b5eec9ef-serving-cert\""
Jul 09 11:42:46.661776 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.661721    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"]
Jul 09 11:42:46.697523 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.697493    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:46.802583 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.802549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0606554b-7f16-435c-aace-6e560212abdc-maas-api-tls\") pod \"maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2\" (UID: \"0606554b-7f16-435c-aace-6e560212abdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:42:46.802810 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.802623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0606554b-7f16-435c-aace-6e560212abdc-openshift-service-ca\") pod \"maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2\" (UID: \"0606554b-7f16-435c-aace-6e560212abdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:42:46.802810 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.802657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw2zf\" (UniqueName: \"kubernetes.io/projected/0606554b-7f16-435c-aace-6e560212abdc-kube-api-access-nw2zf\") pod \"maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2\" (UID: \"0606554b-7f16-435c-aace-6e560212abdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:42:46.838426 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.838326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:46.904108 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.904070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0606554b-7f16-435c-aace-6e560212abdc-openshift-service-ca\") pod \"maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2\" (UID: \"0606554b-7f16-435c-aace-6e560212abdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:42:46.904491 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.904120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw2zf\" (UniqueName: \"kubernetes.io/projected/0606554b-7f16-435c-aace-6e560212abdc-kube-api-access-nw2zf\") pod \"maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2\" (UID: \"0606554b-7f16-435c-aace-6e560212abdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:42:46.904491 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.904206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0606554b-7f16-435c-aace-6e560212abdc-maas-api-tls\") pod \"maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2\" (UID: \"0606554b-7f16-435c-aace-6e560212abdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:42:46.904491 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:42:46.904333    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-serving-cert: secret "maas-api-e2e-mt-b5eec9ef-serving-cert" not found
Jul 09 11:42:46.904491 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:42:46.904407    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0606554b-7f16-435c-aace-6e560212abdc-maas-api-tls podName:0606554b-7f16-435c-aace-6e560212abdc nodeName:}" failed. No retries permitted until 2026-07-09 11:42:47.404385059 +0000 UTC m=+1850.996116207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0606554b-7f16-435c-aace-6e560212abdc-maas-api-tls") pod "maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2" (UID: "0606554b-7f16-435c-aace-6e560212abdc") : secret "maas-api-e2e-mt-b5eec9ef-serving-cert" not found
Jul 09 11:42:46.904826 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.904808    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0606554b-7f16-435c-aace-6e560212abdc-openshift-service-ca\") pod \"maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2\" (UID: \"0606554b-7f16-435c-aace-6e560212abdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:42:46.918104 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.918077    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw2zf\" (UniqueName: \"kubernetes.io/projected/0606554b-7f16-435c-aace-6e560212abdc-kube-api-access-nw2zf\") pod \"maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2\" (UID: \"0606554b-7f16-435c-aace-6e560212abdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:42:46.998741 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:46.998706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:47.313130 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:47.313100    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:47.409118 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:47.409085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0606554b-7f16-435c-aace-6e560212abdc-maas-api-tls\") pod \"maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2\" (UID: \"0606554b-7f16-435c-aace-6e560212abdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:42:47.411552 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:47.411531    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0606554b-7f16-435c-aace-6e560212abdc-maas-api-tls\") pod \"maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2\" (UID: \"0606554b-7f16-435c-aace-6e560212abdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:42:47.414770 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:47.414747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:47.557340 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:47.557291    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:42:47.695198 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:47.695175    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"]
Jul 09 11:42:47.697514 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:42:47.697482    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0606554b_7f16_435c_aace_6e560212abdc.slice/crio-6e85528088c38cc8d94d985b52ed0e2b3bfb4ec3e2b21eb7a201a0410e01f841 WatchSource:0}: Error finding container 6e85528088c38cc8d94d985b52ed0e2b3bfb4ec3e2b21eb7a201a0410e01f841: Status 404 returned error can't find the container with id 6e85528088c38cc8d94d985b52ed0e2b3bfb4ec3e2b21eb7a201a0410e01f841
Jul 09 11:42:48.596623 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:48.596521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2" event={"ID":"0606554b-7f16-435c-aace-6e560212abdc","Type":"ContainerStarted","Data":"b42018989134af2ee839ea5c8fcd076807d5663671d01e3f6914f78f7bf51c13"}
Jul 09 11:42:48.597093 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:48.596625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2" event={"ID":"0606554b-7f16-435c-aace-6e560212abdc","Type":"ContainerStarted","Data":"6e85528088c38cc8d94d985b52ed0e2b3bfb4ec3e2b21eb7a201a0410e01f841"}
Jul 09 11:42:48.597093 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:48.596771    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:42:48.618482 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:48.618432    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2" podStartSLOduration=2.055939207 podStartE2EDuration="2.618416721s" totalImagesPullingTime="562.477514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:42:48.617014128 +0000 UTC m=+1852.208745293" watchObservedRunningTime="2026-07-09 11:42:48.618416721 +0000 UTC m=+1852.210147905"
Jul 09 11:42:51.062659 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:51.062622    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:42:54.608640 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:54.608608    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:42:58.574385 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:42:58.574344    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:43:01.615197 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:01.615169    2563 patch_prober.go:28] interesting pod/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": read tcp 10.134.0.2:58332->10.134.0.35:15021: read: connection reset by peer" start-of-body=
Jul 09 11:43:01.615592 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:01.615236    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs" podUID="61104450-8f60-4486-9f29-717f3148a5fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": read tcp 10.134.0.2:58332->10.134.0.35:15021: read: connection reset by peer"
Jul 09 11:43:01.673616 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:01.673583    2563 generic.go:356] "Generic (PLEG): container finished" podID="61104450-8f60-4486-9f29-717f3148a5fc" containerID="e1ec7c4299d7a55def1915515400844a27d7c48cc0539a4a8dbcb1e70e98a034" exitCode=0
Jul 09 11:43:01.673616 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:01.673624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs" event={"ID":"61104450-8f60-4486-9f29-717f3148a5fc","Type":"ContainerDied","Data":"e1ec7c4299d7a55def1915515400844a27d7c48cc0539a4a8dbcb1e70e98a034"}
Jul 09 11:43:01.674009 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:01.673996    2563 scope.go:117] "RemoveContainer" containerID="e1ec7c4299d7a55def1915515400844a27d7c48cc0539a4a8dbcb1e70e98a034"
Jul 09 11:43:01.674945 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:01.674912    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:43:01.675056 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:01.675000    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:43:01.675056 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:01.675045    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:43:02.680445 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:02.680408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs" event={"ID":"61104450-8f60-4486-9f29-717f3148a5fc","Type":"ContainerStarted","Data":"e4794e25f9c403319532f186d8ea9b30c989893af295f99ceddc6e7df26b9d41"}
Jul 09 11:43:02.894162 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:02.894122    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:43:02.894418 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:02.894392    2563 patch_prober.go:28] interesting pod/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Jul 09 11:43:02.894466 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:02.894443    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs" podUID="61104450-8f60-4486-9f29-717f3148a5fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Jul 09 11:43:03.895127 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:03.895083    2563 patch_prober.go:28] interesting pod/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Jul 09 11:43:03.895534 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:03.895141    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs" podUID="61104450-8f60-4486-9f29-717f3148a5fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Jul 09 11:43:04.894706 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:04.894670    2563 patch_prober.go:28] interesting pod/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Jul 09 11:43:04.894911 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:04.894728    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs" podUID="61104450-8f60-4486-9f29-717f3148a5fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Jul 09 11:43:05.319660 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:05.319624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:43:05.417877 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:05.417843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:43:05.894995 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:05.894960    2563 patch_prober.go:28] interesting pod/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Jul 09 11:43:05.895146 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:05.895044    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs" podUID="61104450-8f60-4486-9f29-717f3148a5fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Jul 09 11:43:06.898240 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:06.898206    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:43:06.898691 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:06.898672    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:43:06.899296 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:06.899278    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:43:08.202805 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:08.199180    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:43:08.328774 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:08.328733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:43:33.759169 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:33.759078    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:43:34.132588 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.132502    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"]
Jul 09 11:43:34.132831 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.132763    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2" podUID="0606554b-7f16-435c-aace-6e560212abdc" containerName="maas-api" containerID="cri-o://b42018989134af2ee839ea5c8fcd076807d5663671d01e3f6914f78f7bf51c13" gracePeriod=30
Jul 09 11:43:34.384654 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.384595    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:43:34.539027 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.538994    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0606554b-7f16-435c-aace-6e560212abdc-openshift-service-ca\") pod \"0606554b-7f16-435c-aace-6e560212abdc\" (UID: \"0606554b-7f16-435c-aace-6e560212abdc\") "
Jul 09 11:43:34.539218 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.539048    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw2zf\" (UniqueName: \"kubernetes.io/projected/0606554b-7f16-435c-aace-6e560212abdc-kube-api-access-nw2zf\") pod \"0606554b-7f16-435c-aace-6e560212abdc\" (UID: \"0606554b-7f16-435c-aace-6e560212abdc\") "
Jul 09 11:43:34.539218 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.539108    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0606554b-7f16-435c-aace-6e560212abdc-maas-api-tls\") pod \"0606554b-7f16-435c-aace-6e560212abdc\" (UID: \"0606554b-7f16-435c-aace-6e560212abdc\") "
Jul 09 11:43:34.539376 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.539352    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0606554b-7f16-435c-aace-6e560212abdc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0606554b-7f16-435c-aace-6e560212abdc" (UID: "0606554b-7f16-435c-aace-6e560212abdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:43:34.541208 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.541178    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0606554b-7f16-435c-aace-6e560212abdc-kube-api-access-nw2zf" (OuterVolumeSpecName: "kube-api-access-nw2zf") pod "0606554b-7f16-435c-aace-6e560212abdc" (UID: "0606554b-7f16-435c-aace-6e560212abdc"). InnerVolumeSpecName "kube-api-access-nw2zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:43:34.541306 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.541271    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0606554b-7f16-435c-aace-6e560212abdc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0606554b-7f16-435c-aace-6e560212abdc" (UID: "0606554b-7f16-435c-aace-6e560212abdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 11:43:34.640233 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.640138    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0606554b-7f16-435c-aace-6e560212abdc-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:43:34.640233 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.640170    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw2zf\" (UniqueName: \"kubernetes.io/projected/0606554b-7f16-435c-aace-6e560212abdc-kube-api-access-nw2zf\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:43:34.640233 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.640182    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0606554b-7f16-435c-aace-6e560212abdc-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:43:34.821514 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.821478    2563 generic.go:356] "Generic (PLEG): container finished" podID="0606554b-7f16-435c-aace-6e560212abdc" containerID="b42018989134af2ee839ea5c8fcd076807d5663671d01e3f6914f78f7bf51c13" exitCode=0
Jul 09 11:43:34.821999 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.821519    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"
Jul 09 11:43:34.821999 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.821520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2" event={"ID":"0606554b-7f16-435c-aace-6e560212abdc","Type":"ContainerDied","Data":"b42018989134af2ee839ea5c8fcd076807d5663671d01e3f6914f78f7bf51c13"}
Jul 09 11:43:34.821999 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.821586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2" event={"ID":"0606554b-7f16-435c-aace-6e560212abdc","Type":"ContainerDied","Data":"6e85528088c38cc8d94d985b52ed0e2b3bfb4ec3e2b21eb7a201a0410e01f841"}
Jul 09 11:43:34.821999 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.821620    2563 scope.go:117] "RemoveContainer" containerID="b42018989134af2ee839ea5c8fcd076807d5663671d01e3f6914f78f7bf51c13"
Jul 09 11:43:34.830367 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.830349    2563 scope.go:117] "RemoveContainer" containerID="b42018989134af2ee839ea5c8fcd076807d5663671d01e3f6914f78f7bf51c13"
Jul 09 11:43:34.830619 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:43:34.830597    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b42018989134af2ee839ea5c8fcd076807d5663671d01e3f6914f78f7bf51c13\": container with ID starting with b42018989134af2ee839ea5c8fcd076807d5663671d01e3f6914f78f7bf51c13 not found: ID does not exist" containerID="b42018989134af2ee839ea5c8fcd076807d5663671d01e3f6914f78f7bf51c13"
Jul 09 11:43:34.830682 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.830627    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b42018989134af2ee839ea5c8fcd076807d5663671d01e3f6914f78f7bf51c13"} err="failed to get container status \"b42018989134af2ee839ea5c8fcd076807d5663671d01e3f6914f78f7bf51c13\": rpc error: code = NotFound desc = could not find container \"b42018989134af2ee839ea5c8fcd076807d5663671d01e3f6914f78f7bf51c13\": container with ID starting with b42018989134af2ee839ea5c8fcd076807d5663671d01e3f6914f78f7bf51c13 not found: ID does not exist"
Jul 09 11:43:34.846143 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.846119    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"]
Jul 09 11:43:34.852036 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.852004    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5eec9ef-57c6c7d8db-jqrs2"]
Jul 09 11:43:34.897960 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.897887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:43:34.950194 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:34.950164    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0606554b-7f16-435c-aace-6e560212abdc" path="/var/lib/kubelet/pods/0606554b-7f16-435c-aace-6e560212abdc/volumes"
Jul 09 11:43:39.861041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:39.861007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:43:44.936936 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:43:44.936898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:21.448911 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:21.448837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:22.281962 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:22.281904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:27.904250 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:27.904218    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:28.010129 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:28.010095    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:28.181003 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:28.180911    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"]
Jul 09 11:44:28.181186 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:28.181109    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs" podUID="61104450-8f60-4486-9f29-717f3148a5fc" containerName="istio-proxy" containerID="cri-o://e4794e25f9c403319532f186d8ea9b30c989893af295f99ceddc6e7df26b9d41" gracePeriod=30
Jul 09 11:44:28.207478 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:28.207441    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:29.520299 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:29.520265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:29.619180 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:29.619144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:29.727791 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:29.727759    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:30.475073 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:30.475028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:30.568476 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:30.568430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:30.680364 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:30.680329    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:32.127351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:32.127316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:33.439700 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.439677    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:44:33.461831 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.461677    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-credential-socket\") pod \"61104450-8f60-4486-9f29-717f3148a5fc\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") "
Jul 09 11:44:33.461831 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.461728    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61104450-8f60-4486-9f29-717f3148a5fc-istiod-ca-cert\") pod \"61104450-8f60-4486-9f29-717f3148a5fc\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") "
Jul 09 11:44:33.461831 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.461763    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-istio-data\") pod \"61104450-8f60-4486-9f29-717f3148a5fc\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") "
Jul 09 11:44:33.461831 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.461823    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-workload-certs\") pod \"61104450-8f60-4486-9f29-717f3148a5fc\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") "
Jul 09 11:44:33.463055 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.461854    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vf4m\" (UniqueName: \"kubernetes.io/projected/61104450-8f60-4486-9f29-717f3148a5fc-kube-api-access-6vf4m\") pod \"61104450-8f60-4486-9f29-717f3148a5fc\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") "
Jul 09 11:44:33.463055 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.461897    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61104450-8f60-4486-9f29-717f3148a5fc-istio-token\") pod \"61104450-8f60-4486-9f29-717f3148a5fc\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") "
Jul 09 11:44:33.463055 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.461920    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-workload-socket\") pod \"61104450-8f60-4486-9f29-717f3148a5fc\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") "
Jul 09 11:44:33.463055 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.461943    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-istio-envoy\") pod \"61104450-8f60-4486-9f29-717f3148a5fc\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") "
Jul 09 11:44:33.463055 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.461996    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61104450-8f60-4486-9f29-717f3148a5fc-istio-podinfo\") pod \"61104450-8f60-4486-9f29-717f3148a5fc\" (UID: \"61104450-8f60-4486-9f29-717f3148a5fc\") "
Jul 09 11:44:33.463055 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.462176    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "61104450-8f60-4486-9f29-717f3148a5fc" (UID: "61104450-8f60-4486-9f29-717f3148a5fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:44:33.463055 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.462357    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "61104450-8f60-4486-9f29-717f3148a5fc" (UID: "61104450-8f60-4486-9f29-717f3148a5fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:44:33.463055 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.463027    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61104450-8f60-4486-9f29-717f3148a5fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "61104450-8f60-4486-9f29-717f3148a5fc" (UID: "61104450-8f60-4486-9f29-717f3148a5fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:44:33.463487 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.463304    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "61104450-8f60-4486-9f29-717f3148a5fc" (UID: "61104450-8f60-4486-9f29-717f3148a5fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:44:33.463487 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.463322    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "61104450-8f60-4486-9f29-717f3148a5fc" (UID: "61104450-8f60-4486-9f29-717f3148a5fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:44:33.465334 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.465308    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "61104450-8f60-4486-9f29-717f3148a5fc" (UID: "61104450-8f60-4486-9f29-717f3148a5fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:44:33.465652 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.465620    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61104450-8f60-4486-9f29-717f3148a5fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "61104450-8f60-4486-9f29-717f3148a5fc" (UID: "61104450-8f60-4486-9f29-717f3148a5fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 11:44:33.465775 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.465756    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61104450-8f60-4486-9f29-717f3148a5fc-kube-api-access-6vf4m" (OuterVolumeSpecName: "kube-api-access-6vf4m") pod "61104450-8f60-4486-9f29-717f3148a5fc" (UID: "61104450-8f60-4486-9f29-717f3148a5fc"). InnerVolumeSpecName "kube-api-access-6vf4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:44:33.465904 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.465888    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61104450-8f60-4486-9f29-717f3148a5fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "61104450-8f60-4486-9f29-717f3148a5fc" (UID: "61104450-8f60-4486-9f29-717f3148a5fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:44:33.562591 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.562551    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:44:33.562591 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.562584    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:44:33.562591 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.562593    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61104450-8f60-4486-9f29-717f3148a5fc-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:44:33.562591 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.562601    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:44:33.562896 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.562611    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61104450-8f60-4486-9f29-717f3148a5fc-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:44:33.562896 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.562621    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:44:33.562896 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.562648    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61104450-8f60-4486-9f29-717f3148a5fc-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:44:33.562896 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.562656    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vf4m\" (UniqueName: \"kubernetes.io/projected/61104450-8f60-4486-9f29-717f3148a5fc-kube-api-access-6vf4m\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:44:33.562896 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:33.562665    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61104450-8f60-4486-9f29-717f3148a5fc-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:44:34.077231 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:34.077196    2563 generic.go:356] "Generic (PLEG): container finished" podID="61104450-8f60-4486-9f29-717f3148a5fc" containerID="e4794e25f9c403319532f186d8ea9b30c989893af295f99ceddc6e7df26b9d41" exitCode=0
Jul 09 11:44:34.077427 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:34.077243    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"
Jul 09 11:44:34.077427 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:34.077255    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs" event={"ID":"61104450-8f60-4486-9f29-717f3148a5fc","Type":"ContainerDied","Data":"e4794e25f9c403319532f186d8ea9b30c989893af295f99ceddc6e7df26b9d41"}
Jul 09 11:44:34.077427 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:34.077307    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs" event={"ID":"61104450-8f60-4486-9f29-717f3148a5fc","Type":"ContainerDied","Data":"6be4ff5d0c161a625acd57c2ef92f498e70a672cb6743dcb286cc74209b0a0a1"}
Jul 09 11:44:34.078107 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:34.077335    2563 scope.go:117] "RemoveContainer" containerID="e4794e25f9c403319532f186d8ea9b30c989893af295f99ceddc6e7df26b9d41"
Jul 09 11:44:34.090499 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:34.090477    2563 scope.go:117] "RemoveContainer" containerID="e1ec7c4299d7a55def1915515400844a27d7c48cc0539a4a8dbcb1e70e98a034"
Jul 09 11:44:34.098047 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:34.098025    2563 scope.go:117] "RemoveContainer" containerID="e4794e25f9c403319532f186d8ea9b30c989893af295f99ceddc6e7df26b9d41"
Jul 09 11:44:34.098304 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:44:34.098285    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4794e25f9c403319532f186d8ea9b30c989893af295f99ceddc6e7df26b9d41\": container with ID starting with e4794e25f9c403319532f186d8ea9b30c989893af295f99ceddc6e7df26b9d41 not found: ID does not exist" containerID="e4794e25f9c403319532f186d8ea9b30c989893af295f99ceddc6e7df26b9d41"
Jul 09 11:44:34.098353 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:34.098312    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4794e25f9c403319532f186d8ea9b30c989893af295f99ceddc6e7df26b9d41"} err="failed to get container status \"e4794e25f9c403319532f186d8ea9b30c989893af295f99ceddc6e7df26b9d41\": rpc error: code = NotFound desc = could not find container \"e4794e25f9c403319532f186d8ea9b30c989893af295f99ceddc6e7df26b9d41\": container with ID starting with e4794e25f9c403319532f186d8ea9b30c989893af295f99ceddc6e7df26b9d41 not found: ID does not exist"
Jul 09 11:44:34.098353 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:34.098329    2563 scope.go:117] "RemoveContainer" containerID="e1ec7c4299d7a55def1915515400844a27d7c48cc0539a4a8dbcb1e70e98a034"
Jul 09 11:44:34.098566 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:44:34.098549    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ec7c4299d7a55def1915515400844a27d7c48cc0539a4a8dbcb1e70e98a034\": container with ID starting with e1ec7c4299d7a55def1915515400844a27d7c48cc0539a4a8dbcb1e70e98a034 not found: ID does not exist" containerID="e1ec7c4299d7a55def1915515400844a27d7c48cc0539a4a8dbcb1e70e98a034"
Jul 09 11:44:34.098607 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:34.098573    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ec7c4299d7a55def1915515400844a27d7c48cc0539a4a8dbcb1e70e98a034"} err="failed to get container status \"e1ec7c4299d7a55def1915515400844a27d7c48cc0539a4a8dbcb1e70e98a034\": rpc error: code = NotFound desc = could not find container \"e1ec7c4299d7a55def1915515400844a27d7c48cc0539a4a8dbcb1e70e98a034\": container with ID starting with e1ec7c4299d7a55def1915515400844a27d7c48cc0539a4a8dbcb1e70e98a034 not found: ID does not exist"
Jul 09 11:44:34.104552 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:34.104528    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"]
Jul 09 11:44:34.113610 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:34.113586    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b5eec9ef-openshift-default-c964cb6dc-x6mrs"]
Jul 09 11:44:34.949917 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:34.949884    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61104450-8f60-4486-9f29-717f3148a5fc" path="/var/lib/kubelet/pods/61104450-8f60-4486-9f29-717f3148a5fc/volumes"
Jul 09 11:44:38.687776 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:38.687744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:38.783253 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:38.783214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:39.398087 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:39.398051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:45.786351 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:45.786314    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:45.854204 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:45.854167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:45.964395 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:45.964358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:46.062498 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:46.062397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:56.529526 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:56.529490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:44:56.891978 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:44:56.891865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:17.906936 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:17.906899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:23.824533 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:23.824490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:23.882916 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:23.882879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:24.181212 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.181111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:24.471831 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.471730    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"]
Jul 09 11:45:24.472147 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.472121    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61104450-8f60-4486-9f29-717f3148a5fc" containerName="istio-proxy"
Jul 09 11:45:24.472147 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.472144    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="61104450-8f60-4486-9f29-717f3148a5fc" containerName="istio-proxy"
Jul 09 11:45:24.472339 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.472161    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0606554b-7f16-435c-aace-6e560212abdc" containerName="maas-api"
Jul 09 11:45:24.472339 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.472169    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0606554b-7f16-435c-aace-6e560212abdc" containerName="maas-api"
Jul 09 11:45:24.472339 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.472179    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61104450-8f60-4486-9f29-717f3148a5fc" containerName="istio-proxy"
Jul 09 11:45:24.472339 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.472187    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="61104450-8f60-4486-9f29-717f3148a5fc" containerName="istio-proxy"
Jul 09 11:45:24.472339 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.472266    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="61104450-8f60-4486-9f29-717f3148a5fc" containerName="istio-proxy"
Jul 09 11:45:24.472339 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.472285    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0606554b-7f16-435c-aace-6e560212abdc" containerName="maas-api"
Jul 09 11:45:24.475333 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.475313    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.477985 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.477965    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-944007-openshift-default-dockercfg-vxsfp\""
Jul 09 11:45:24.481396 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.481370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-token\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.481519 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.481430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-podinfo\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.481519 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.481483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpcb6\" (UniqueName: \"kubernetes.io/projected/6a1f8f42-587b-405b-aefa-9cd01741cca9-kube-api-access-gpcb6\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.482007 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.481589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-data\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.482007 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.481675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-workload-socket\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.482007 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.481718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-envoy\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.482007 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.481757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1f8f42-587b-405b-aefa-9cd01741cca9-istiod-ca-cert\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.482007 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.481826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-credential-socket\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.482007 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.481866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-workload-certs\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.489548 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.489527    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"]
Jul 09 11:45:24.499901 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.499879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:24.582657 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.582624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-token\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.582858 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.582673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-podinfo\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.582858 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.582694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpcb6\" (UniqueName: \"kubernetes.io/projected/6a1f8f42-587b-405b-aefa-9cd01741cca9-kube-api-access-gpcb6\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.582858 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.582730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-data\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.582858 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.582764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-workload-socket\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.582858 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.582806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-envoy\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.582858 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.582831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1f8f42-587b-405b-aefa-9cd01741cca9-istiod-ca-cert\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.583238 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.582869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-credential-socket\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.583238 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.582894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-workload-certs\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.583343 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.583301    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-data\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.583343 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.583317    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-workload-certs\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.583485 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.583468    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-credential-socket\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.583597 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.583572    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-workload-socket\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.583714 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.583691    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1f8f42-587b-405b-aefa-9cd01741cca9-istiod-ca-cert\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.585225 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.585194    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-envoy\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.585341 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.585322    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-podinfo\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.591428 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.591403    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-token\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.591617 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.591595    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpcb6\" (UniqueName: \"kubernetes.io/projected/6a1f8f42-587b-405b-aefa-9cd01741cca9-kube-api-access-gpcb6\") pod \"e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") " pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.785951 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.785919    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:24.787311 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.787286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:24.920722 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.920558    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"]
Jul 09 11:45:24.923307 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:45:24.923279    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1f8f42_587b_405b_aefa_9cd01741cca9.slice/crio-2ce4a3ceb1c9bee7d1afa3acce2cd19265cfd6a21768d3db849a0a6af6a07302 WatchSource:0}: Error finding container 2ce4a3ceb1c9bee7d1afa3acce2cd19265cfd6a21768d3db849a0a6af6a07302: Status 404 returned error can't find the container with id 2ce4a3ceb1c9bee7d1afa3acce2cd19265cfd6a21768d3db849a0a6af6a07302
Jul 09 11:45:24.925572 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.925536    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:45:24.925686 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.925616    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:45:24.925686 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:24.925661    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:45:25.301176 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:25.301133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz" event={"ID":"6a1f8f42-587b-405b-aefa-9cd01741cca9","Type":"ContainerStarted","Data":"99c4cb343568f5c56d3655e1cb463b9668ff8c9ca5f1bc7335775f352f2857e7"}
Jul 09 11:45:25.301343 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:25.301193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz" event={"ID":"6a1f8f42-587b-405b-aefa-9cd01741cca9","Type":"ContainerStarted","Data":"2ce4a3ceb1c9bee7d1afa3acce2cd19265cfd6a21768d3db849a0a6af6a07302"}
Jul 09 11:45:25.322276 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:25.322226    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz" podStartSLOduration=1.322212841 podStartE2EDuration="1.322212841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:45:25.320673319 +0000 UTC m=+2008.912404506" watchObservedRunningTime="2026-07-09 11:45:25.322212841 +0000 UTC m=+2008.913944005"
Jul 09 11:45:25.786273 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:25.786240    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:25.791024 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:25.791001    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:26.308939 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:26.308907    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:26.312178 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:26.312147    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:27.292800 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.292734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:27.403663 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.403628    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"]
Jul 09 11:45:27.404068 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.403990    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="61104450-8f60-4486-9f29-717f3148a5fc" containerName="istio-proxy"
Jul 09 11:45:27.407155 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.407133    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:45:27.409813 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.409771    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-944007-serving-cert\""
Jul 09 11:45:27.421522 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.421499    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"]
Jul 09 11:45:27.449751 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.449720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:27.506639 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.506593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nds4m\" (UniqueName: \"kubernetes.io/projected/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-kube-api-access-nds4m\") pod \"maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x\" (UID: \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:45:27.506825 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.506645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-openshift-service-ca\") pod \"maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x\" (UID: \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:45:27.506825 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.506722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-maas-api-tls\") pod \"maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x\" (UID: \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:45:27.607699 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.607593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nds4m\" (UniqueName: \"kubernetes.io/projected/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-kube-api-access-nds4m\") pod \"maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x\" (UID: \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:45:27.607699 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.607654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-openshift-service-ca\") pod \"maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x\" (UID: \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:45:27.607945 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.607701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-maas-api-tls\") pod \"maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x\" (UID: \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:45:27.607945 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:45:27.607908    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-serving-cert: secret "maas-api-e2e-inf-a-944007-serving-cert" not found
Jul 09 11:45:27.608127 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:45:27.607994    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-maas-api-tls podName:8e7a14b1-4a66-4f45-bcc1-1c3c52125169 nodeName:}" failed. No retries permitted until 2026-07-09 11:45:28.107961554 +0000 UTC m=+2011.699692703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-maas-api-tls") pod "maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x" (UID: "8e7a14b1-4a66-4f45-bcc1-1c3c52125169") : secret "maas-api-e2e-inf-a-944007-serving-cert" not found
Jul 09 11:45:27.608497 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.608474    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-openshift-service-ca\") pod \"maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x\" (UID: \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:45:27.609276 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.609251    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:27.636989 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.636952    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nds4m\" (UniqueName: \"kubernetes.io/projected/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-kube-api-access-nds4m\") pod \"maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x\" (UID: \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:45:27.765220 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:27.765185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:28.108524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:28.108488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:28.111934 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:28.111898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-maas-api-tls\") pod \"maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x\" (UID: \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:45:28.114525 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:28.114501    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-maas-api-tls\") pod \"maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x\" (UID: \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:45:28.206478 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:28.206445    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:28.317227 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:28.317199    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:45:28.454189 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:28.454159    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"]
Jul 09 11:45:28.456172 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:45:28.456137    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7a14b1_4a66_4f45_bcc1_1c3c52125169.slice/crio-4da6e34d0e9528c7ff7c291a6723caad50e546b43e58193916648b934e9c52f2 WatchSource:0}: Error finding container 4da6e34d0e9528c7ff7c291a6723caad50e546b43e58193916648b934e9c52f2: Status 404 returned error can't find the container with id 4da6e34d0e9528c7ff7c291a6723caad50e546b43e58193916648b934e9c52f2
Jul 09 11:45:28.457960 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:28.457942    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 11:45:29.323111 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:29.323077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x" event={"ID":"8e7a14b1-4a66-4f45-bcc1-1c3c52125169","Type":"ContainerStarted","Data":"f4b6421dcfb2a731a4a1bdc2f91f168d9d78425884c3c7f76e31b827db0eac1b"}
Jul 09 11:45:29.323414 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:29.323130    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x" event={"ID":"8e7a14b1-4a66-4f45-bcc1-1c3c52125169","Type":"ContainerStarted","Data":"4da6e34d0e9528c7ff7c291a6723caad50e546b43e58193916648b934e9c52f2"}
Jul 09 11:45:29.323414 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:29.323166    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:45:29.345829 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:29.345770    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x" podStartSLOduration=1.888812986 podStartE2EDuration="2.345753272s" totalImagesPullingTime="456.940286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:45:29.343387638 +0000 UTC m=+2012.935118814" watchObservedRunningTime="2026-07-09 11:45:29.345753272 +0000 UTC m=+2012.937484437"
Jul 09 11:45:31.802993 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:31.802957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:32.132843 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.132734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:32.447937 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.447854    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"]
Jul 09 11:45:32.451608 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.451588    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.454619 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.454598    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0d824f-openshift-default-dockercfg-4gxvl\""
Jul 09 11:45:32.488684 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.488658    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"]
Jul 09 11:45:32.541541 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.541499    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:32.546703 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.546665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b648c063-b5ea-404c-aad2-01af26178b97-istio-token\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.546885 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.546710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tsvs\" (UniqueName: \"kubernetes.io/projected/b648c063-b5ea-404c-aad2-01af26178b97-kube-api-access-5tsvs\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.546885 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.546771    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b648c063-b5ea-404c-aad2-01af26178b97-istio-podinfo\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.546885 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.546830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-istio-data\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.546885 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.546865    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-workload-socket\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.547078 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.546889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-istio-envoy\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.547078 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.546940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-credential-socket\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.547078 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.546974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-workload-certs\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.547078 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.547058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b648c063-b5ea-404c-aad2-01af26178b97-istiod-ca-cert\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.647930 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.647890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b648c063-b5ea-404c-aad2-01af26178b97-istio-token\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.647930 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.647931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tsvs\" (UniqueName: \"kubernetes.io/projected/b648c063-b5ea-404c-aad2-01af26178b97-kube-api-access-5tsvs\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.648139 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.648054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b648c063-b5ea-404c-aad2-01af26178b97-istio-podinfo\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.648139 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.648103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-istio-data\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.648249 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.648138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-workload-socket\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.648249 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.648155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-istio-envoy\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.648249 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.648215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-credential-socket\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.648387 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.648254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-workload-certs\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.648387 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.648312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b648c063-b5ea-404c-aad2-01af26178b97-istiod-ca-cert\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.648672 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.648644    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-istio-data\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.648806 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.648688    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-workload-socket\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.648887 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.648867    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-credential-socket\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.648951 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.648910    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-workload-certs\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.649132 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.649117    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b648c063-b5ea-404c-aad2-01af26178b97-istiod-ca-cert\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.650655 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.650634    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b648c063-b5ea-404c-aad2-01af26178b97-istio-podinfo\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.650856 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.650835    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-istio-envoy\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.657503 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.657468    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tsvs\" (UniqueName: \"kubernetes.io/projected/b648c063-b5ea-404c-aad2-01af26178b97-kube-api-access-5tsvs\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.657889 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.657867    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b648c063-b5ea-404c-aad2-01af26178b97-istio-token\") pod \"e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") " pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.764986 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.764955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:32.788805 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.788764    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:32.902414 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.902386    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"]
Jul 09 11:45:32.904446 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:45:32.904413    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb648c063_b5ea_404c_aad2_01af26178b97.slice/crio-36cc376c692a8907765dad14719e2e0a0026de3b1996f97f6490aa43daf9bb13 WatchSource:0}: Error finding container 36cc376c692a8907765dad14719e2e0a0026de3b1996f97f6490aa43daf9bb13: Status 404 returned error can't find the container with id 36cc376c692a8907765dad14719e2e0a0026de3b1996f97f6490aa43daf9bb13
Jul 09 11:45:32.907017 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.906988    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:45:32.907081 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.907056    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:45:32.907122 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:32.907084    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:45:33.342794 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:33.342748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8" event={"ID":"b648c063-b5ea-404c-aad2-01af26178b97","Type":"ContainerStarted","Data":"8670cede2c2f707ea992b801d4f8284ed8df945a2dddcea68f386976fdca8955"}
Jul 09 11:45:33.342966 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:33.342812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8" event={"ID":"b648c063-b5ea-404c-aad2-01af26178b97","Type":"ContainerStarted","Data":"36cc376c692a8907765dad14719e2e0a0026de3b1996f97f6490aa43daf9bb13"}
Jul 09 11:45:33.364663 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:33.364613    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8" podStartSLOduration=1.364597486 podStartE2EDuration="1.364597486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:45:33.363270003 +0000 UTC m=+2016.955001169" watchObservedRunningTime="2026-07-09 11:45:33.364597486 +0000 UTC m=+2016.956328650"
Jul 09 11:45:33.765999 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:33.765962    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:33.770521 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:33.770496    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:34.347162 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:34.347127    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:34.348211 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:34.348187    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:35.333223 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.333189    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:45:35.376869 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.376829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:35.476633 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.476601    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"]
Jul 09 11:45:35.486053 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.486027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:45:35.486615 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.486571    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:35.488419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.488396    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0d824f-serving-cert\""
Jul 09 11:45:35.494579 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.494240    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"]
Jul 09 11:45:35.578201 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.578167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0d824f-88987d875-qmbrj\" (UID: \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:45:35.578415 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.578211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6dv8\" (UniqueName: \"kubernetes.io/projected/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-kube-api-access-j6dv8\") pod \"maas-api-e2e-inf-b-0d824f-88987d875-qmbrj\" (UID: \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:45:35.578415 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.578249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d824f-88987d875-qmbrj\" (UID: \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:45:35.633898 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.633765    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:35.679659 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.679624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0d824f-88987d875-qmbrj\" (UID: \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:45:35.679883 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.679674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6dv8\" (UniqueName: \"kubernetes.io/projected/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-kube-api-access-j6dv8\") pod \"maas-api-e2e-inf-b-0d824f-88987d875-qmbrj\" (UID: \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:45:35.679883 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.679708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d824f-88987d875-qmbrj\" (UID: \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:45:35.679883 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:45:35.679879    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-serving-cert: secret "maas-api-e2e-inf-b-0d824f-serving-cert" not found
Jul 09 11:45:35.680069 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:45:35.679936    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-maas-api-tls podName:99a697d6-2f38-4aa7-8aed-c7f847dc3c16 nodeName:}" failed. No retries permitted until 2026-07-09 11:45:36.1799171 +0000 UTC m=+2019.771648259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-maas-api-tls") pod "maas-api-e2e-inf-b-0d824f-88987d875-qmbrj" (UID: "99a697d6-2f38-4aa7-8aed-c7f847dc3c16") : secret "maas-api-e2e-inf-b-0d824f-serving-cert" not found
Jul 09 11:45:35.680404 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.680377    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0d824f-88987d875-qmbrj\" (UID: \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:45:35.692759 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.692733    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6dv8\" (UniqueName: \"kubernetes.io/projected/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-kube-api-access-j6dv8\") pod \"maas-api-e2e-inf-b-0d824f-88987d875-qmbrj\" (UID: \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:45:35.946450 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:35.946353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:36.185889 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:36.185848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d824f-88987d875-qmbrj\" (UID: \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:45:36.188416 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:36.188392    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d824f-88987d875-qmbrj\" (UID: \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:45:36.293838 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:36.293806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:36.394433 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:36.394396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:36.398487 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:36.398465    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:45:36.552965 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:36.551354    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"]
Jul 09 11:45:36.560070 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:45:36.558069    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a697d6_2f38_4aa7_8aed_c7f847dc3c16.slice/crio-25ae5a9105b2ec34f8e0447f2045322b8a78b025bccf0ea66fa465d05231cf4a WatchSource:0}: Error finding container 25ae5a9105b2ec34f8e0447f2045322b8a78b025bccf0ea66fa465d05231cf4a: Status 404 returned error can't find the container with id 25ae5a9105b2ec34f8e0447f2045322b8a78b025bccf0ea66fa465d05231cf4a
Jul 09 11:45:37.364654 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:37.364609    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj" event={"ID":"99a697d6-2f38-4aa7-8aed-c7f847dc3c16","Type":"ContainerStarted","Data":"60afe2f58c946a0cc51e30c0b615023643736b7f86cd68844e2a255decfe63ac"}
Jul 09 11:45:37.364892 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:37.364830    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:45:37.364892 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:37.364873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj" event={"ID":"99a697d6-2f38-4aa7-8aed-c7f847dc3c16","Type":"ContainerStarted","Data":"25ae5a9105b2ec34f8e0447f2045322b8a78b025bccf0ea66fa465d05231cf4a"}
Jul 09 11:45:37.384965 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:37.384915    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj" podStartSLOduration=1.961023435 podStartE2EDuration="2.384900211s" totalImagesPullingTime="423.876776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:45:37.382911784 +0000 UTC m=+2020.974642974" watchObservedRunningTime="2026-07-09 11:45:37.384900211 +0000 UTC m=+2020.976631374"
Jul 09 11:45:39.890799 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:39.890747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:42.349995 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:42.349946    2563 patch_prober.go:28] interesting pod/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 09 11:45:42.350411 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:42.350014    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz" podUID="6a1f8f42-587b-405b-aefa-9cd01741cca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 11:45:43.374627 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:43.374594    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:45:43.394015 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:43.393975    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a1f8f42-587b-405b-aefa-9cd01741cca9" containerID="99c4cb343568f5c56d3655e1cb463b9668ff8c9ca5f1bc7335775f352f2857e7" exitCode=0
Jul 09 11:45:43.394162 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:43.394036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz" event={"ID":"6a1f8f42-587b-405b-aefa-9cd01741cca9","Type":"ContainerDied","Data":"99c4cb343568f5c56d3655e1cb463b9668ff8c9ca5f1bc7335775f352f2857e7"}
Jul 09 11:45:43.394521 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:43.394504    2563 scope.go:117] "RemoveContainer" containerID="99c4cb343568f5c56d3655e1cb463b9668ff8c9ca5f1bc7335775f352f2857e7"
Jul 09 11:45:43.395552 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:43.395528    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:45:43.395603 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:43.395590    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:45:43.395641 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:43.395620    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:45:44.399847 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:44.399814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz" event={"ID":"6a1f8f42-587b-405b-aefa-9cd01741cca9","Type":"ContainerStarted","Data":"fdd32fb4753a9fe6714b92ffd5c7f1ebeeaf5d92912aa5bfd0a01bf5e3c1d387"}
Jul 09 11:45:44.787328 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:44.787281    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:44.787553 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:44.787423    2563 patch_prober.go:28] interesting pod/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 09 11:45:44.787553 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:44.787469    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz" podUID="6a1f8f42-587b-405b-aefa-9cd01741cca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 09 11:45:45.786627 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:45.786577    2563 patch_prober.go:28] interesting pod/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 09 11:45:45.787184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:45.786650    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz" podUID="6a1f8f42-587b-405b-aefa-9cd01741cca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 09 11:45:46.786514 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:46.786463    2563 patch_prober.go:28] interesting pod/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 09 11:45:46.786706 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:46.786537    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz" podUID="6a1f8f42-587b-405b-aefa-9cd01741cca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 09 11:45:47.787093 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:47.787053    2563 patch_prober.go:28] interesting pod/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 09 11:45:47.787489 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:47.787118    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz" podUID="6a1f8f42-587b-405b-aefa-9cd01741cca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 09 11:45:48.790426 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:48.790395    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:48.790833 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:48.790682    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:48.791391 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:48.791369    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:45:49.033889 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:49.033849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:49.135013 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:49.134933    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:49.990482 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:49.989967    2563 patch_prober.go:28] interesting pod/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": read tcp 10.134.0.2:41860->10.134.0.39:15021: read: connection reset by peer" start-of-body=
Jul 09 11:45:49.990482 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:49.990032    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8" podUID="b648c063-b5ea-404c-aad2-01af26178b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": read tcp 10.134.0.2:41860->10.134.0.39:15021: read: connection reset by peer"
Jul 09 11:45:50.427164 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:50.427133    2563 generic.go:356] "Generic (PLEG): container finished" podID="b648c063-b5ea-404c-aad2-01af26178b97" containerID="8670cede2c2f707ea992b801d4f8284ed8df945a2dddcea68f386976fdca8955" exitCode=0
Jul 09 11:45:50.427362 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:50.427213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8" event={"ID":"b648c063-b5ea-404c-aad2-01af26178b97","Type":"ContainerDied","Data":"8670cede2c2f707ea992b801d4f8284ed8df945a2dddcea68f386976fdca8955"}
Jul 09 11:45:50.427710 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:50.427696    2563 scope.go:117] "RemoveContainer" containerID="8670cede2c2f707ea992b801d4f8284ed8df945a2dddcea68f386976fdca8955"
Jul 09 11:45:50.428603 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:50.428579    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:45:50.428682 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:50.428665    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:45:50.428731 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:50.428715    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:45:51.434958 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:51.434924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8" event={"ID":"b648c063-b5ea-404c-aad2-01af26178b97","Type":"ContainerStarted","Data":"ba4ada90a77b1740d28e79ad593b6b6cf23f857fe75ec1be933819d27493b9bf"}
Jul 09 11:45:51.765426 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:51.765382    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:51.765619 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:51.765595    2563 patch_prober.go:28] interesting pod/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Jul 09 11:45:51.765665 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:51.765647    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8" podUID="b648c063-b5ea-404c-aad2-01af26178b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Jul 09 11:45:52.765869 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:52.765818    2563 patch_prober.go:28] interesting pod/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Jul 09 11:45:52.766388 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:52.765895    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8" podUID="b648c063-b5ea-404c-aad2-01af26178b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Jul 09 11:45:53.766055 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:53.766018    2563 patch_prober.go:28] interesting pod/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Jul 09 11:45:53.766473 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:53.766089    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8" podUID="b648c063-b5ea-404c-aad2-01af26178b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Jul 09 11:45:54.022233 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:54.022142    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:54.765406 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:54.765364    2563 patch_prober.go:28] interesting pod/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Jul 09 11:45:54.765587 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:54.765443    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8" podUID="b648c063-b5ea-404c-aad2-01af26178b97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Jul 09 11:45:55.768773 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:55.768744    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:55.769206 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:55.769025    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:55.769608 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:55.769589    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:45:58.509063 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:58.509029    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:45:58.603527 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:45:58.603495    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:46:15.969668 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:15.969636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:46:18.128104 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:18.128022    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:46:18.228260 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:18.228231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:46:48.880051 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:48.880018    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:46:49.869073 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:49.869038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:46:50.037255 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:50.037221    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:46:50.420129 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:50.420096    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:46:50.585625 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:50.585592    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:46:50.683897 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:50.683751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:46:50.932939 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:50.932899    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"]
Jul 09 11:46:50.933116 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:50.933097    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x" podUID="8e7a14b1-4a66-4f45-bcc1-1c3c52125169" containerName="maas-api" containerID="cri-o://f4b6421dcfb2a731a4a1bdc2f91f168d9d78425884c3c7f76e31b827db0eac1b" gracePeriod=30
Jul 09 11:46:50.951727 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:50.951704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:46:51.183469 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.183444    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:46:51.286602 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.286571    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-openshift-service-ca\") pod \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\" (UID: \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\") "
Jul 09 11:46:51.286821 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.286608    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-maas-api-tls\") pod \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\" (UID: \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\") "
Jul 09 11:46:51.286821 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.286692    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nds4m\" (UniqueName: \"kubernetes.io/projected/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-kube-api-access-nds4m\") pod \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\" (UID: \"8e7a14b1-4a66-4f45-bcc1-1c3c52125169\") "
Jul 09 11:46:51.286985 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.286961    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e7a14b1-4a66-4f45-bcc1-1c3c52125169" (UID: "8e7a14b1-4a66-4f45-bcc1-1c3c52125169"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:46:51.288755 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.288730    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e7a14b1-4a66-4f45-bcc1-1c3c52125169" (UID: "8e7a14b1-4a66-4f45-bcc1-1c3c52125169"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 11:46:51.288872 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.288769    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-kube-api-access-nds4m" (OuterVolumeSpecName: "kube-api-access-nds4m") pod "8e7a14b1-4a66-4f45-bcc1-1c3c52125169" (UID: "8e7a14b1-4a66-4f45-bcc1-1c3c52125169"). InnerVolumeSpecName "kube-api-access-nds4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:46:51.392824 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.387680    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nds4m\" (UniqueName: \"kubernetes.io/projected/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-kube-api-access-nds4m\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:46:51.392824 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.387725    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:46:51.392824 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.387745    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e7a14b1-4a66-4f45-bcc1-1c3c52125169-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:46:51.726310 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.726215    2563 generic.go:356] "Generic (PLEG): container finished" podID="8e7a14b1-4a66-4f45-bcc1-1c3c52125169" containerID="f4b6421dcfb2a731a4a1bdc2f91f168d9d78425884c3c7f76e31b827db0eac1b" exitCode=0
Jul 09 11:46:51.726310 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.726270    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"
Jul 09 11:46:51.726531 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.726313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x" event={"ID":"8e7a14b1-4a66-4f45-bcc1-1c3c52125169","Type":"ContainerDied","Data":"f4b6421dcfb2a731a4a1bdc2f91f168d9d78425884c3c7f76e31b827db0eac1b"}
Jul 09 11:46:51.726531 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.726389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x" event={"ID":"8e7a14b1-4a66-4f45-bcc1-1c3c52125169","Type":"ContainerDied","Data":"4da6e34d0e9528c7ff7c291a6723caad50e546b43e58193916648b934e9c52f2"}
Jul 09 11:46:51.726531 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.726413    2563 scope.go:117] "RemoveContainer" containerID="f4b6421dcfb2a731a4a1bdc2f91f168d9d78425884c3c7f76e31b827db0eac1b"
Jul 09 11:46:51.736120 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.736099    2563 scope.go:117] "RemoveContainer" containerID="f4b6421dcfb2a731a4a1bdc2f91f168d9d78425884c3c7f76e31b827db0eac1b"
Jul 09 11:46:51.736431 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:46:51.736411    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b6421dcfb2a731a4a1bdc2f91f168d9d78425884c3c7f76e31b827db0eac1b\": container with ID starting with f4b6421dcfb2a731a4a1bdc2f91f168d9d78425884c3c7f76e31b827db0eac1b not found: ID does not exist" containerID="f4b6421dcfb2a731a4a1bdc2f91f168d9d78425884c3c7f76e31b827db0eac1b"
Jul 09 11:46:51.736500 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.736439    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b6421dcfb2a731a4a1bdc2f91f168d9d78425884c3c7f76e31b827db0eac1b"} err="failed to get container status \"f4b6421dcfb2a731a4a1bdc2f91f168d9d78425884c3c7f76e31b827db0eac1b\": rpc error: code = NotFound desc = could not find container \"f4b6421dcfb2a731a4a1bdc2f91f168d9d78425884c3c7f76e31b827db0eac1b\": container with ID starting with f4b6421dcfb2a731a4a1bdc2f91f168d9d78425884c3c7f76e31b827db0eac1b not found: ID does not exist"
Jul 09 11:46:51.751835 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.751805    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"]
Jul 09 11:46:51.758125 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:51.758102    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-944007-b89f7cb78-g4d9x"]
Jul 09 11:46:52.950202 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:52.950161    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e7a14b1-4a66-4f45-bcc1-1c3c52125169" path="/var/lib/kubelet/pods/8e7a14b1-4a66-4f45-bcc1-1c3c52125169/volumes"
Jul 09 11:46:55.487318 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:55.487281    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:46:56.951548 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:56.951521    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:46:56.956954 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:46:56.956932    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:47:23.515451 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:23.515418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:32.906344 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:32.906302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:33.162411 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:33.162329    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"]
Jul 09 11:47:33.162593 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:33.162565    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz" podUID="6a1f8f42-587b-405b-aefa-9cd01741cca9" containerName="istio-proxy" containerID="cri-o://fdd32fb4753a9fe6714b92ffd5c7f1ebeeaf5d92912aa5bfd0a01bf5e3c1d387" gracePeriod=30
Jul 09 11:47:33.177263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:33.177223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:33.595155 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:33.595122    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:33.793327 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:33.793291    2563 patch_prober.go:28] interesting pod/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 09 11:47:33.793492 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:33.793351    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz" podUID="6a1f8f42-587b-405b-aefa-9cd01741cca9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 11:47:33.966559 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:33.966476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:34.088653 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.088613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:34.167071 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.167033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:34.385158 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.385119    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"]
Jul 09 11:47:34.385397 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.385370    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj" podUID="99a697d6-2f38-4aa7-8aed-c7f847dc3c16" containerName="maas-api" containerID="cri-o://60afe2f58c946a0cc51e30c0b615023643736b7f86cd68844e2a255decfe63ac" gracePeriod=30
Jul 09 11:47:34.453880 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.453806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:34.634358 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.634333    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:47:34.687713 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.687631    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-openshift-service-ca\") pod \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\" (UID: \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\") "
Jul 09 11:47:34.687713 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.687665    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6dv8\" (UniqueName: \"kubernetes.io/projected/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-kube-api-access-j6dv8\") pod \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\" (UID: \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\") "
Jul 09 11:47:34.687942 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.687822    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-maas-api-tls\") pod \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\" (UID: \"99a697d6-2f38-4aa7-8aed-c7f847dc3c16\") "
Jul 09 11:47:34.688049 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.688028    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "99a697d6-2f38-4aa7-8aed-c7f847dc3c16" (UID: "99a697d6-2f38-4aa7-8aed-c7f847dc3c16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:47:34.688168 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.688144    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:34.689823 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.689796    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "99a697d6-2f38-4aa7-8aed-c7f847dc3c16" (UID: "99a697d6-2f38-4aa7-8aed-c7f847dc3c16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 11:47:34.689938 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.689912    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-kube-api-access-j6dv8" (OuterVolumeSpecName: "kube-api-access-j6dv8") pod "99a697d6-2f38-4aa7-8aed-c7f847dc3c16" (UID: "99a697d6-2f38-4aa7-8aed-c7f847dc3c16"). InnerVolumeSpecName "kube-api-access-j6dv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:47:34.788916 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.788882    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:34.788916 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.788912    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6dv8\" (UniqueName: \"kubernetes.io/projected/99a697d6-2f38-4aa7-8aed-c7f847dc3c16-kube-api-access-j6dv8\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:34.919558 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.919518    2563 generic.go:356] "Generic (PLEG): container finished" podID="99a697d6-2f38-4aa7-8aed-c7f847dc3c16" containerID="60afe2f58c946a0cc51e30c0b615023643736b7f86cd68844e2a255decfe63ac" exitCode=0
Jul 09 11:47:34.919732 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.919567    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"
Jul 09 11:47:34.919732 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.919567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj" event={"ID":"99a697d6-2f38-4aa7-8aed-c7f847dc3c16","Type":"ContainerDied","Data":"60afe2f58c946a0cc51e30c0b615023643736b7f86cd68844e2a255decfe63ac"}
Jul 09 11:47:34.919732 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.919717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj" event={"ID":"99a697d6-2f38-4aa7-8aed-c7f847dc3c16","Type":"ContainerDied","Data":"25ae5a9105b2ec34f8e0447f2045322b8a78b025bccf0ea66fa465d05231cf4a"}
Jul 09 11:47:34.919964 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.919745    2563 scope.go:117] "RemoveContainer" containerID="60afe2f58c946a0cc51e30c0b615023643736b7f86cd68844e2a255decfe63ac"
Jul 09 11:47:34.928264 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.928248    2563 scope.go:117] "RemoveContainer" containerID="60afe2f58c946a0cc51e30c0b615023643736b7f86cd68844e2a255decfe63ac"
Jul 09 11:47:34.928538 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:47:34.928521    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60afe2f58c946a0cc51e30c0b615023643736b7f86cd68844e2a255decfe63ac\": container with ID starting with 60afe2f58c946a0cc51e30c0b615023643736b7f86cd68844e2a255decfe63ac not found: ID does not exist" containerID="60afe2f58c946a0cc51e30c0b615023643736b7f86cd68844e2a255decfe63ac"
Jul 09 11:47:34.928599 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.928549    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60afe2f58c946a0cc51e30c0b615023643736b7f86cd68844e2a255decfe63ac"} err="failed to get container status \"60afe2f58c946a0cc51e30c0b615023643736b7f86cd68844e2a255decfe63ac\": rpc error: code = NotFound desc = could not find container \"60afe2f58c946a0cc51e30c0b615023643736b7f86cd68844e2a255decfe63ac\": container with ID starting with 60afe2f58c946a0cc51e30c0b615023643736b7f86cd68844e2a255decfe63ac not found: ID does not exist"
Jul 09 11:47:34.949952 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.949887    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"]
Jul 09 11:47:34.953620 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:34.953597    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d824f-88987d875-qmbrj"]
Jul 09 11:47:36.950479 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:36.950448    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99a697d6-2f38-4aa7-8aed-c7f847dc3c16" path="/var/lib/kubelet/pods/99a697d6-2f38-4aa7-8aed-c7f847dc3c16/volumes"
Jul 09 11:47:38.418764 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.418736    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:47:38.520754 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.520721    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-data\") pod \"6a1f8f42-587b-405b-aefa-9cd01741cca9\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") "
Jul 09 11:47:38.520949 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.520773    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-envoy\") pod \"6a1f8f42-587b-405b-aefa-9cd01741cca9\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") "
Jul 09 11:47:38.520949 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.520863    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1f8f42-587b-405b-aefa-9cd01741cca9-istiod-ca-cert\") pod \"6a1f8f42-587b-405b-aefa-9cd01741cca9\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") "
Jul 09 11:47:38.520949 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.520896    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-workload-certs\") pod \"6a1f8f42-587b-405b-aefa-9cd01741cca9\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") "
Jul 09 11:47:38.520949 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.520926    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-podinfo\") pod \"6a1f8f42-587b-405b-aefa-9cd01741cca9\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") "
Jul 09 11:47:38.520949 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.520945    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-credential-socket\") pod \"6a1f8f42-587b-405b-aefa-9cd01741cca9\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") "
Jul 09 11:47:38.521217 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.521046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-token\") pod \"6a1f8f42-587b-405b-aefa-9cd01741cca9\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") "
Jul 09 11:47:38.521217 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.521082    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpcb6\" (UniqueName: \"kubernetes.io/projected/6a1f8f42-587b-405b-aefa-9cd01741cca9-kube-api-access-gpcb6\") pod \"6a1f8f42-587b-405b-aefa-9cd01741cca9\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") "
Jul 09 11:47:38.521217 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.521112    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-workload-socket\") pod \"6a1f8f42-587b-405b-aefa-9cd01741cca9\" (UID: \"6a1f8f42-587b-405b-aefa-9cd01741cca9\") "
Jul 09 11:47:38.521369 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.521218    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6a1f8f42-587b-405b-aefa-9cd01741cca9" (UID: "6a1f8f42-587b-405b-aefa-9cd01741cca9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:47:38.521369 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.521234    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6a1f8f42-587b-405b-aefa-9cd01741cca9" (UID: "6a1f8f42-587b-405b-aefa-9cd01741cca9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:47:38.521369 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.521290    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1f8f42-587b-405b-aefa-9cd01741cca9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6a1f8f42-587b-405b-aefa-9cd01741cca9" (UID: "6a1f8f42-587b-405b-aefa-9cd01741cca9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:47:38.521531 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.521446    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6a1f8f42-587b-405b-aefa-9cd01741cca9" (UID: "6a1f8f42-587b-405b-aefa-9cd01741cca9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:47:38.521593 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.521555    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:38.521593 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.521573    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1f8f42-587b-405b-aefa-9cd01741cca9-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:38.521593 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.521587    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:38.521770 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.521601    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:38.521770 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.521597    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-data" (OuterVolumeSpecName: "istio-data") pod "6a1f8f42-587b-405b-aefa-9cd01741cca9" (UID: "6a1f8f42-587b-405b-aefa-9cd01741cca9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:47:38.523266 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.523239    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1f8f42-587b-405b-aefa-9cd01741cca9-kube-api-access-gpcb6" (OuterVolumeSpecName: "kube-api-access-gpcb6") pod "6a1f8f42-587b-405b-aefa-9cd01741cca9" (UID: "6a1f8f42-587b-405b-aefa-9cd01741cca9"). InnerVolumeSpecName "kube-api-access-gpcb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:47:38.523361 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.523303    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-token" (OuterVolumeSpecName: "istio-token") pod "6a1f8f42-587b-405b-aefa-9cd01741cca9" (UID: "6a1f8f42-587b-405b-aefa-9cd01741cca9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:47:38.523480 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.523462    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6a1f8f42-587b-405b-aefa-9cd01741cca9" (UID: "6a1f8f42-587b-405b-aefa-9cd01741cca9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 11:47:38.523536 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.523470    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6a1f8f42-587b-405b-aefa-9cd01741cca9" (UID: "6a1f8f42-587b-405b-aefa-9cd01741cca9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:47:38.622742 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.622702    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:38.622742 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.622733    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:38.622742 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.622745    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpcb6\" (UniqueName: \"kubernetes.io/projected/6a1f8f42-587b-405b-aefa-9cd01741cca9-kube-api-access-gpcb6\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:38.623003 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.622756    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:38.623003 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.622768    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1f8f42-587b-405b-aefa-9cd01741cca9-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:38.938223 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.938131    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a1f8f42-587b-405b-aefa-9cd01741cca9" containerID="fdd32fb4753a9fe6714b92ffd5c7f1ebeeaf5d92912aa5bfd0a01bf5e3c1d387" exitCode=0
Jul 09 11:47:38.938223 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.938191    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"
Jul 09 11:47:38.938426 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.938224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz" event={"ID":"6a1f8f42-587b-405b-aefa-9cd01741cca9","Type":"ContainerDied","Data":"fdd32fb4753a9fe6714b92ffd5c7f1ebeeaf5d92912aa5bfd0a01bf5e3c1d387"}
Jul 09 11:47:38.938426 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.938281    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz" event={"ID":"6a1f8f42-587b-405b-aefa-9cd01741cca9","Type":"ContainerDied","Data":"2ce4a3ceb1c9bee7d1afa3acce2cd19265cfd6a21768d3db849a0a6af6a07302"}
Jul 09 11:47:38.938426 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.938314    2563 scope.go:117] "RemoveContainer" containerID="fdd32fb4753a9fe6714b92ffd5c7f1ebeeaf5d92912aa5bfd0a01bf5e3c1d387"
Jul 09 11:47:38.947245 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.947227    2563 scope.go:117] "RemoveContainer" containerID="99c4cb343568f5c56d3655e1cb463b9668ff8c9ca5f1bc7335775f352f2857e7"
Jul 09 11:47:38.955978 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.955960    2563 scope.go:117] "RemoveContainer" containerID="fdd32fb4753a9fe6714b92ffd5c7f1ebeeaf5d92912aa5bfd0a01bf5e3c1d387"
Jul 09 11:47:38.956262 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:47:38.956239    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd32fb4753a9fe6714b92ffd5c7f1ebeeaf5d92912aa5bfd0a01bf5e3c1d387\": container with ID starting with fdd32fb4753a9fe6714b92ffd5c7f1ebeeaf5d92912aa5bfd0a01bf5e3c1d387 not found: ID does not exist" containerID="fdd32fb4753a9fe6714b92ffd5c7f1ebeeaf5d92912aa5bfd0a01bf5e3c1d387"
Jul 09 11:47:38.956335 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.956275    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd32fb4753a9fe6714b92ffd5c7f1ebeeaf5d92912aa5bfd0a01bf5e3c1d387"} err="failed to get container status \"fdd32fb4753a9fe6714b92ffd5c7f1ebeeaf5d92912aa5bfd0a01bf5e3c1d387\": rpc error: code = NotFound desc = could not find container \"fdd32fb4753a9fe6714b92ffd5c7f1ebeeaf5d92912aa5bfd0a01bf5e3c1d387\": container with ID starting with fdd32fb4753a9fe6714b92ffd5c7f1ebeeaf5d92912aa5bfd0a01bf5e3c1d387 not found: ID does not exist"
Jul 09 11:47:38.956335 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.956299    2563 scope.go:117] "RemoveContainer" containerID="99c4cb343568f5c56d3655e1cb463b9668ff8c9ca5f1bc7335775f352f2857e7"
Jul 09 11:47:38.956564 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:47:38.956548    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c4cb343568f5c56d3655e1cb463b9668ff8c9ca5f1bc7335775f352f2857e7\": container with ID starting with 99c4cb343568f5c56d3655e1cb463b9668ff8c9ca5f1bc7335775f352f2857e7 not found: ID does not exist" containerID="99c4cb343568f5c56d3655e1cb463b9668ff8c9ca5f1bc7335775f352f2857e7"
Jul 09 11:47:38.956623 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.956570    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c4cb343568f5c56d3655e1cb463b9668ff8c9ca5f1bc7335775f352f2857e7"} err="failed to get container status \"99c4cb343568f5c56d3655e1cb463b9668ff8c9ca5f1bc7335775f352f2857e7\": rpc error: code = NotFound desc = could not find container \"99c4cb343568f5c56d3655e1cb463b9668ff8c9ca5f1bc7335775f352f2857e7\": container with ID starting with 99c4cb343568f5c56d3655e1cb463b9668ff8c9ca5f1bc7335775f352f2857e7 not found: ID does not exist"
Jul 09 11:47:38.964143 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.964122    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"]
Jul 09 11:47:38.972232 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:38.972212    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-944007-openshift-default-86d9cb54f-sjrvz"]
Jul 09 11:47:39.974979 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:39.974947    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:40.076343 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:40.076299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:40.240908 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:40.240832    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"]
Jul 09 11:47:40.241094 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:40.241062    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8" podUID="b648c063-b5ea-404c-aad2-01af26178b97" containerName="istio-proxy" containerID="cri-o://ba4ada90a77b1740d28e79ad593b6b6cf23f857fe75ec1be933819d27493b9bf" gracePeriod=30
Jul 09 11:47:40.277768 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:40.277736    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:40.771011 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:40.770973    2563 patch_prober.go:28] interesting pod/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 09 11:47:40.771211 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:40.771052    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8" podUID="b648c063-b5ea-404c-aad2-01af26178b97" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 11:47:40.874689 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:40.874654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:40.949944 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:40.949914    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a1f8f42-587b-405b-aefa-9cd01741cca9" path="/var/lib/kubelet/pods/6a1f8f42-587b-405b-aefa-9cd01741cca9/volumes"
Jul 09 11:47:41.180423 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.180344    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"]
Jul 09 11:47:41.180772 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.180677    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1f8f42-587b-405b-aefa-9cd01741cca9" containerName="istio-proxy"
Jul 09 11:47:41.180772 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.180691    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1f8f42-587b-405b-aefa-9cd01741cca9" containerName="istio-proxy"
Jul 09 11:47:41.180772 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.180706    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1f8f42-587b-405b-aefa-9cd01741cca9" containerName="istio-proxy"
Jul 09 11:47:41.180772 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.180711    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1f8f42-587b-405b-aefa-9cd01741cca9" containerName="istio-proxy"
Jul 09 11:47:41.180772 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.180720    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7a14b1-4a66-4f45-bcc1-1c3c52125169" containerName="maas-api"
Jul 09 11:47:41.180772 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.180725    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7a14b1-4a66-4f45-bcc1-1c3c52125169" containerName="maas-api"
Jul 09 11:47:41.180772 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.180735    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99a697d6-2f38-4aa7-8aed-c7f847dc3c16" containerName="maas-api"
Jul 09 11:47:41.180772 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.180741    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="99a697d6-2f38-4aa7-8aed-c7f847dc3c16" containerName="maas-api"
Jul 09 11:47:41.181045 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.180826    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="99a697d6-2f38-4aa7-8aed-c7f847dc3c16" containerName="maas-api"
Jul 09 11:47:41.181045 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.180838    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a1f8f42-587b-405b-aefa-9cd01741cca9" containerName="istio-proxy"
Jul 09 11:47:41.181045 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.180846    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a1f8f42-587b-405b-aefa-9cd01741cca9" containerName="istio-proxy"
Jul 09 11:47:41.181045 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.180852    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7a14b1-4a66-4f45-bcc1-1c3c52125169" containerName="maas-api"
Jul 09 11:47:41.185185 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.185166    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.187834 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.187812    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-21f543-openshift-default-dockercfg-ln5pn\""
Jul 09 11:47:41.196737 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.196715    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"]
Jul 09 11:47:41.212912 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.212866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:41.244715 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.244678    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9683206a-413f-4cb0-a97a-97bbf729bc02-istiod-ca-cert\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.244919 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.244729    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-workload-socket\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.244919 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.244819    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-token\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.244919 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.244882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-envoy\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.244919 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.244906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-data\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.245082 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.244949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-podinfo\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.245082 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.244978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-credential-socket\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.245082 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.245040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-workload-certs\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.245082 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.245079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbbzc\" (UniqueName: \"kubernetes.io/projected/9683206a-413f-4cb0-a97a-97bbf729bc02-kube-api-access-xbbzc\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.345990 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.345951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9683206a-413f-4cb0-a97a-97bbf729bc02-istiod-ca-cert\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.345990 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.345993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-workload-socket\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.346236 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.346019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-token\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.346236 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.346071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-envoy\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.346236 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.346094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-data\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.346236 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.346137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-podinfo\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.346236 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.346165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-credential-socket\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.346236 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.346203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-workload-certs\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.346236 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.346230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbbzc\" (UniqueName: \"kubernetes.io/projected/9683206a-413f-4cb0-a97a-97bbf729bc02-kube-api-access-xbbzc\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.346591 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.346458    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-workload-socket\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.346591 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.346511    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-data\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.346762 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.346741    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-credential-socket\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.346863 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.346802    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-workload-certs\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.346863 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.346811    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9683206a-413f-4cb0-a97a-97bbf729bc02-istiod-ca-cert\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.348506 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.348474    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-envoy\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.348794 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.348769    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-podinfo\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.356116 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.356093    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-token\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.356450 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.356414    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbbzc\" (UniqueName: \"kubernetes.io/projected/9683206a-413f-4cb0-a97a-97bbf729bc02-kube-api-access-xbbzc\") pod \"e2e-api-a-21f543-openshift-default-54699669cc-v57xp\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") " pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.496429 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.496344    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:41.533193 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.533161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:41.642339 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.642315    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"]
Jul 09 11:47:41.644367 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:47:41.644340    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9683206a_413f_4cb0_a97a_97bbf729bc02.slice/crio-d95f7e6510905257664114f1f5f5fcd8893bc3b645de5a8f3ddf4ec5d727d676 WatchSource:0}: Error finding container d95f7e6510905257664114f1f5f5fcd8893bc3b645de5a8f3ddf4ec5d727d676: Status 404 returned error can't find the container with id d95f7e6510905257664114f1f5f5fcd8893bc3b645de5a8f3ddf4ec5d727d676
Jul 09 11:47:41.646525 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.646486    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:47:41.646610 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.646570    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:47:41.646654 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.646609    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:47:41.954078 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.954041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp" event={"ID":"9683206a-413f-4cb0-a97a-97bbf729bc02","Type":"ContainerStarted","Data":"c6775ef44f636ac83b0261c8d3c021f5f2ddd89b661ed2cace17421f749b7a97"}
Jul 09 11:47:41.954258 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.954093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp" event={"ID":"9683206a-413f-4cb0-a97a-97bbf729bc02","Type":"ContainerStarted","Data":"d95f7e6510905257664114f1f5f5fcd8893bc3b645de5a8f3ddf4ec5d727d676"}
Jul 09 11:47:41.978516 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:41.978452    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp" podStartSLOduration=0.978431045 podStartE2EDuration="978.431045ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:47:41.975450135 +0000 UTC m=+2145.567181320" watchObservedRunningTime="2026-07-09 11:47:41.978431045 +0000 UTC m=+2145.570162210"
Jul 09 11:47:42.497294 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:42.497251    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:42.502343 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:42.502317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:42.958515 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:42.958485    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:42.959562 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:42.959544    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:47:43.913590 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:43.913555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:43.974103 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:43.974069    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"]
Jul 09 11:47:43.977872 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:43.977851    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:47:43.980362 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:43.980339    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-21f543-serving-cert\""
Jul 09 11:47:44.000813 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:43.992687    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"]
Jul 09 11:47:44.000813 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.000594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:44.064856 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.064817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:44.070492 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.070440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-openshift-service-ca\") pod \"maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk\" (UID: \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:47:44.070667 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.070565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-maas-api-tls\") pod \"maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk\" (UID: \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:47:44.070667 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.070595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8knd\" (UniqueName: \"kubernetes.io/projected/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-kube-api-access-h8knd\") pod \"maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk\" (UID: \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:47:44.171971 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.171870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-openshift-service-ca\") pod \"maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk\" (UID: \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:47:44.171971 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.171924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-maas-api-tls\") pod \"maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk\" (UID: \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:47:44.172220 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:47:44.172028    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-serving-cert: secret "maas-api-e2e-api-a-21f543-serving-cert" not found
Jul 09 11:47:44.172220 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:47:44.172087    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-maas-api-tls podName:5751ba9d-88ad-4e47-9cbc-3fd7834185e1 nodeName:}" failed. No retries permitted until 2026-07-09 11:47:44.672069277 +0000 UTC m=+2148.263800418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-maas-api-tls") pod "maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk" (UID: "5751ba9d-88ad-4e47-9cbc-3fd7834185e1") : secret "maas-api-e2e-api-a-21f543-serving-cert" not found
Jul 09 11:47:44.172220 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.172083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8knd\" (UniqueName: \"kubernetes.io/projected/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-kube-api-access-h8knd\") pod \"maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk\" (UID: \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:47:44.172663 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.172638    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-openshift-service-ca\") pod \"maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk\" (UID: \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:47:44.184673 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.184649    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8knd\" (UniqueName: \"kubernetes.io/projected/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-kube-api-access-h8knd\") pod \"maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk\" (UID: \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:47:44.369650 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.369610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:44.572985 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.572938    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:44.672847 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.672806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:44.678040 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.678017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-maas-api-tls\") pod \"maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk\" (UID: \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:47:44.680593 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.680565    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-maas-api-tls\") pod \"maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk\" (UID: \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:47:44.890669 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:44.890571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:47:45.026991 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.026962    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"]
Jul 09 11:47:45.028862 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:47:45.028830    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5751ba9d_88ad_4e47_9cbc_3fd7834185e1.slice/crio-c9bb39a024a4bc69eb107aca99afb065fa96dc0098d49430a9812100fe4b9001 WatchSource:0}: Error finding container c9bb39a024a4bc69eb107aca99afb065fa96dc0098d49430a9812100fe4b9001: Status 404 returned error can't find the container with id c9bb39a024a4bc69eb107aca99afb065fa96dc0098d49430a9812100fe4b9001
Jul 09 11:47:45.498626 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.498599    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:47:45.585211 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.585186    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-credential-socket\") pod \"b648c063-b5ea-404c-aad2-01af26178b97\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") "
Jul 09 11:47:45.585357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.585236    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-istio-envoy\") pod \"b648c063-b5ea-404c-aad2-01af26178b97\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") "
Jul 09 11:47:45.585357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.585264    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b648c063-b5ea-404c-aad2-01af26178b97-istio-podinfo\") pod \"b648c063-b5ea-404c-aad2-01af26178b97\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") "
Jul 09 11:47:45.585357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.585287    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-workload-socket\") pod \"b648c063-b5ea-404c-aad2-01af26178b97\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") "
Jul 09 11:47:45.585357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.585313    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b648c063-b5ea-404c-aad2-01af26178b97-istiod-ca-cert\") pod \"b648c063-b5ea-404c-aad2-01af26178b97\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") "
Jul 09 11:47:45.585357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.585346    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tsvs\" (UniqueName: \"kubernetes.io/projected/b648c063-b5ea-404c-aad2-01af26178b97-kube-api-access-5tsvs\") pod \"b648c063-b5ea-404c-aad2-01af26178b97\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") "
Jul 09 11:47:45.585659 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.585437    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-istio-data\") pod \"b648c063-b5ea-404c-aad2-01af26178b97\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") "
Jul 09 11:47:45.585659 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.585461    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-workload-certs\") pod \"b648c063-b5ea-404c-aad2-01af26178b97\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") "
Jul 09 11:47:45.585659 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.585484    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b648c063-b5ea-404c-aad2-01af26178b97" (UID: "b648c063-b5ea-404c-aad2-01af26178b97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:47:45.585659 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.585498    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b648c063-b5ea-404c-aad2-01af26178b97-istio-token\") pod \"b648c063-b5ea-404c-aad2-01af26178b97\" (UID: \"b648c063-b5ea-404c-aad2-01af26178b97\") "
Jul 09 11:47:45.585659 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.585621    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b648c063-b5ea-404c-aad2-01af26178b97" (UID: "b648c063-b5ea-404c-aad2-01af26178b97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:47:45.585950 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.585815    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b648c063-b5ea-404c-aad2-01af26178b97-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b648c063-b5ea-404c-aad2-01af26178b97" (UID: "b648c063-b5ea-404c-aad2-01af26178b97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:47:45.585950 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.585904    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:45.585950 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.585924    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:45.585950 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.585941    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b648c063-b5ea-404c-aad2-01af26178b97-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:45.586318 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.586065    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b648c063-b5ea-404c-aad2-01af26178b97" (UID: "b648c063-b5ea-404c-aad2-01af26178b97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:47:45.586384 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.586352    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-istio-data" (OuterVolumeSpecName: "istio-data") pod "b648c063-b5ea-404c-aad2-01af26178b97" (UID: "b648c063-b5ea-404c-aad2-01af26178b97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:47:45.587384 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.587357    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b648c063-b5ea-404c-aad2-01af26178b97-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b648c063-b5ea-404c-aad2-01af26178b97" (UID: "b648c063-b5ea-404c-aad2-01af26178b97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 11:47:45.587471 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.587407    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b648c063-b5ea-404c-aad2-01af26178b97" (UID: "b648c063-b5ea-404c-aad2-01af26178b97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:47:45.587522 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.587510    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b648c063-b5ea-404c-aad2-01af26178b97-kube-api-access-5tsvs" (OuterVolumeSpecName: "kube-api-access-5tsvs") pod "b648c063-b5ea-404c-aad2-01af26178b97" (UID: "b648c063-b5ea-404c-aad2-01af26178b97"). InnerVolumeSpecName "kube-api-access-5tsvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:47:45.587825 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.587809    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b648c063-b5ea-404c-aad2-01af26178b97-istio-token" (OuterVolumeSpecName: "istio-token") pod "b648c063-b5ea-404c-aad2-01af26178b97" (UID: "b648c063-b5ea-404c-aad2-01af26178b97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:47:45.687337 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.687300    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b648c063-b5ea-404c-aad2-01af26178b97-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:45.687337 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.687336    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tsvs\" (UniqueName: \"kubernetes.io/projected/b648c063-b5ea-404c-aad2-01af26178b97-kube-api-access-5tsvs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:45.687548 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.687350    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:45.687548 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.687362    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:45.687548 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.687373    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b648c063-b5ea-404c-aad2-01af26178b97-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:45.687548 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.687383    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b648c063-b5ea-404c-aad2-01af26178b97-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:47:45.976002 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.975884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk" event={"ID":"5751ba9d-88ad-4e47-9cbc-3fd7834185e1","Type":"ContainerStarted","Data":"e3a1695a2066daa96bd00b22be1466ed991246a3c9f2488d07b2442e04e42002"}
Jul 09 11:47:45.976002 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.975962    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:47:45.976002 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.975990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk" event={"ID":"5751ba9d-88ad-4e47-9cbc-3fd7834185e1","Type":"ContainerStarted","Data":"c9bb39a024a4bc69eb107aca99afb065fa96dc0098d49430a9812100fe4b9001"}
Jul 09 11:47:45.977551 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.977525    2563 generic.go:356] "Generic (PLEG): container finished" podID="b648c063-b5ea-404c-aad2-01af26178b97" containerID="ba4ada90a77b1740d28e79ad593b6b6cf23f857fe75ec1be933819d27493b9bf" exitCode=0
Jul 09 11:47:45.977687 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.977567    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"
Jul 09 11:47:45.977687 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.977572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8" event={"ID":"b648c063-b5ea-404c-aad2-01af26178b97","Type":"ContainerDied","Data":"ba4ada90a77b1740d28e79ad593b6b6cf23f857fe75ec1be933819d27493b9bf"}
Jul 09 11:47:45.977687 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.977617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8" event={"ID":"b648c063-b5ea-404c-aad2-01af26178b97","Type":"ContainerDied","Data":"36cc376c692a8907765dad14719e2e0a0026de3b1996f97f6490aa43daf9bb13"}
Jul 09 11:47:45.977687 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.977647    2563 scope.go:117] "RemoveContainer" containerID="ba4ada90a77b1740d28e79ad593b6b6cf23f857fe75ec1be933819d27493b9bf"
Jul 09 11:47:45.986403 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.986385    2563 scope.go:117] "RemoveContainer" containerID="8670cede2c2f707ea992b801d4f8284ed8df945a2dddcea68f386976fdca8955"
Jul 09 11:47:45.994851 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.994829    2563 scope.go:117] "RemoveContainer" containerID="ba4ada90a77b1740d28e79ad593b6b6cf23f857fe75ec1be933819d27493b9bf"
Jul 09 11:47:45.995147 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:47:45.995128    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba4ada90a77b1740d28e79ad593b6b6cf23f857fe75ec1be933819d27493b9bf\": container with ID starting with ba4ada90a77b1740d28e79ad593b6b6cf23f857fe75ec1be933819d27493b9bf not found: ID does not exist" containerID="ba4ada90a77b1740d28e79ad593b6b6cf23f857fe75ec1be933819d27493b9bf"
Jul 09 11:47:45.995205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.995156    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba4ada90a77b1740d28e79ad593b6b6cf23f857fe75ec1be933819d27493b9bf"} err="failed to get container status \"ba4ada90a77b1740d28e79ad593b6b6cf23f857fe75ec1be933819d27493b9bf\": rpc error: code = NotFound desc = could not find container \"ba4ada90a77b1740d28e79ad593b6b6cf23f857fe75ec1be933819d27493b9bf\": container with ID starting with ba4ada90a77b1740d28e79ad593b6b6cf23f857fe75ec1be933819d27493b9bf not found: ID does not exist"
Jul 09 11:47:45.995205 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.995176    2563 scope.go:117] "RemoveContainer" containerID="8670cede2c2f707ea992b801d4f8284ed8df945a2dddcea68f386976fdca8955"
Jul 09 11:47:45.995394 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:47:45.995377    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8670cede2c2f707ea992b801d4f8284ed8df945a2dddcea68f386976fdca8955\": container with ID starting with 8670cede2c2f707ea992b801d4f8284ed8df945a2dddcea68f386976fdca8955 not found: ID does not exist" containerID="8670cede2c2f707ea992b801d4f8284ed8df945a2dddcea68f386976fdca8955"
Jul 09 11:47:45.995435 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.995398    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8670cede2c2f707ea992b801d4f8284ed8df945a2dddcea68f386976fdca8955"} err="failed to get container status \"8670cede2c2f707ea992b801d4f8284ed8df945a2dddcea68f386976fdca8955\": rpc error: code = NotFound desc = could not find container \"8670cede2c2f707ea992b801d4f8284ed8df945a2dddcea68f386976fdca8955\": container with ID starting with 8670cede2c2f707ea992b801d4f8284ed8df945a2dddcea68f386976fdca8955 not found: ID does not exist"
Jul 09 11:47:45.995610 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:45.995571    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk" podStartSLOduration=2.49572992 podStartE2EDuration="2.995556277s" totalImagesPullingTime="499.826357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:47:45.993829121 +0000 UTC m=+2149.585560319" watchObservedRunningTime="2026-07-09 11:47:45.995556277 +0000 UTC m=+2149.587287441"
Jul 09 11:47:46.010214 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:46.010177    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"]
Jul 09 11:47:46.014688 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:46.014659    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0d824f-openshift-default-6f68d7c9f-9pkc8"]
Jul 09 11:47:46.952822 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:46.952731    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b648c063-b5ea-404c-aad2-01af26178b97" path="/var/lib/kubelet/pods/b648c063-b5ea-404c-aad2-01af26178b97/volumes"
Jul 09 11:47:48.413824 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:48.413773    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:48.952159 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:48.952121    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:49.210766 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.210693    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"]
Jul 09 11:47:49.211055 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.211034    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b648c063-b5ea-404c-aad2-01af26178b97" containerName="istio-proxy"
Jul 09 11:47:49.211055 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.211049    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b648c063-b5ea-404c-aad2-01af26178b97" containerName="istio-proxy"
Jul 09 11:47:49.211214 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.211065    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b648c063-b5ea-404c-aad2-01af26178b97" containerName="istio-proxy"
Jul 09 11:47:49.211214 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.211074    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b648c063-b5ea-404c-aad2-01af26178b97" containerName="istio-proxy"
Jul 09 11:47:49.211214 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.211127    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b648c063-b5ea-404c-aad2-01af26178b97" containerName="istio-proxy"
Jul 09 11:47:49.211214 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.211136    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b648c063-b5ea-404c-aad2-01af26178b97" containerName="istio-proxy"
Jul 09 11:47:49.215559 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.215529    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.218129 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.218110    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e9b58c-openshift-default-dockercfg-mqxph\""
Jul 09 11:47:49.226883 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.226863    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"]
Jul 09 11:47:49.239761 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.239732    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:49.321940 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.321896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-workload-certs\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.321940 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.321940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-envoy\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.322158 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.322025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-credential-socket\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.322158 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.322081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-token\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.322158 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.322103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-podinfo\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.322291 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.322158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-data\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.322291 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.322181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istiod-ca-cert\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.322291 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.322205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-workload-socket\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.322291 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.322241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nwp8\" (UniqueName: \"kubernetes.io/projected/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-kube-api-access-8nwp8\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.423639 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.423595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-workload-socket\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.424199 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.423649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nwp8\" (UniqueName: \"kubernetes.io/projected/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-kube-api-access-8nwp8\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.424199 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.423694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-workload-certs\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.424199 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.423713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-envoy\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.424199 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.423742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-credential-socket\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.424199 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.423818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-token\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.424199 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.423845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-podinfo\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.424199 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.423904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-data\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.424199 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.423928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istiod-ca-cert\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.424199 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.424086    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-workload-socket\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.425051 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.424263    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-workload-certs\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.425051 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.424520    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-credential-socket\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.425051 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.424974    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-data\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.425662 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.425640    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istiod-ca-cert\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.426419 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.426401    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-envoy\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.426550 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.426523    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-podinfo\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.434052 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.433985    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-token\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.434169 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.434068    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nwp8\" (UniqueName: \"kubernetes.io/projected/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-kube-api-access-8nwp8\") pod \"e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") " pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.472670 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.472590    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:49.526964 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.526930    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:49.669291 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.665730    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"]
Jul 09 11:47:49.670569 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.670532    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:47:49.670689 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.670606    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:47:49.670689 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.670646    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:47:49.997311 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.997270    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks" event={"ID":"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f","Type":"ContainerStarted","Data":"7891a492c4e9be91469e930ab72fb08ece640b2933f624b66c7a2433c0fe66cc"}
Jul 09 11:47:49.997516 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:49.997330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks" event={"ID":"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f","Type":"ContainerStarted","Data":"7ebc001444c88f5d6da1fd94a7f8e7e5bdbbb78c184f2981fffbbc397db84475"}
Jul 09 11:47:50.019093 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:50.019043    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks" podStartSLOduration=1.019026678 podStartE2EDuration="1.019026678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:47:50.017834499 +0000 UTC m=+2153.609565667" watchObservedRunningTime="2026-07-09 11:47:50.019026678 +0000 UTC m=+2153.610757841"
Jul 09 11:47:50.527364 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:50.527328    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:50.532336 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:50.532301    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:51.001661 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:51.001635    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:51.002592 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:51.002571    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:47:51.963683 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:51.963647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:51.988921 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:51.988896    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:47:52.045524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.045467    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"]
Jul 09 11:47:52.049362 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.049337    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:47:52.054920 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.054893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e9b58c-serving-cert\""
Jul 09 11:47:52.061930 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.061908    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"]
Jul 09 11:47:52.070097 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.070068    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:52.151004 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.150964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-maas-api-tls\") pod \"maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch\" (UID: \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:47:52.151336 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.151305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbffm\" (UniqueName: \"kubernetes.io/projected/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-kube-api-access-kbffm\") pod \"maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch\" (UID: \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:47:52.151437 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.151387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-openshift-service-ca\") pod \"maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch\" (UID: \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:47:52.162942 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.162907    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:52.252871 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.252843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-openshift-service-ca\") pod \"maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch\" (UID: \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:47:52.253048 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.252924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-maas-api-tls\") pod \"maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch\" (UID: \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:47:52.253048 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.252962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbffm\" (UniqueName: \"kubernetes.io/projected/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-kube-api-access-kbffm\") pod \"maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch\" (UID: \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:47:52.253173 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:47:52.253055    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-serving-cert: secret "maas-api-e2e-api-b-e9b58c-serving-cert" not found
Jul 09 11:47:52.253173 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:47:52.253115    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-maas-api-tls podName:1a6a89eb-d786-4f71-a1f0-407fee5c9a51 nodeName:}" failed. No retries permitted until 2026-07-09 11:47:52.753098431 +0000 UTC m=+2156.344829574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-maas-api-tls") pod "maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch" (UID: "1a6a89eb-d786-4f71-a1f0-407fee5c9a51") : secret "maas-api-e2e-api-b-e9b58c-serving-cert" not found
Jul 09 11:47:52.253506 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.253488    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-openshift-service-ca\") pod \"maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch\" (UID: \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:47:52.265079 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.265058    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbffm\" (UniqueName: \"kubernetes.io/projected/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-kube-api-access-kbffm\") pod \"maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch\" (UID: \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:47:52.367253 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.367214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:52.601580 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.601485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:52.698238 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.698207    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:52.758035 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.757988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-maas-api-tls\") pod \"maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch\" (UID: \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:47:52.761427 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.761390    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-maas-api-tls\") pod \"maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch\" (UID: \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:47:52.963389 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:52.963270    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:47:53.121776 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:53.121750    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"]
Jul 09 11:47:53.124014 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:47:53.123974    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6a89eb_d786_4f71_a1f0_407fee5c9a51.slice/crio-6046873df4b49dfc8b92e7ae5687c6fd455191373e018e7bdb1dc415d4e1f262 WatchSource:0}: Error finding container 6046873df4b49dfc8b92e7ae5687c6fd455191373e018e7bdb1dc415d4e1f262: Status 404 returned error can't find the container with id 6046873df4b49dfc8b92e7ae5687c6fd455191373e018e7bdb1dc415d4e1f262
Jul 09 11:47:54.015848 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:54.015813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch" event={"ID":"1a6a89eb-d786-4f71-a1f0-407fee5c9a51","Type":"ContainerStarted","Data":"1e6f045e095396bf55e8dc7bb81f65c22a0822aa48a603a45fab7c7ca78712de"}
Jul 09 11:47:54.016079 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:54.015870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch" event={"ID":"1a6a89eb-d786-4f71-a1f0-407fee5c9a51","Type":"ContainerStarted","Data":"6046873df4b49dfc8b92e7ae5687c6fd455191373e018e7bdb1dc415d4e1f262"}
Jul 09 11:47:54.016079 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:54.015897    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:47:54.036451 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:54.036390    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch" podStartSLOduration=1.713458624 podStartE2EDuration="2.03637209s" totalImagesPullingTime="322.913466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:47:54.033959378 +0000 UTC m=+2157.625690542" watchObservedRunningTime="2026-07-09 11:47:54.03637209 +0000 UTC m=+2157.628103253"
Jul 09 11:47:56.468825 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:56.468775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:47:59.001023 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:59.000977    2563 patch_prober.go:28] interesting pod/e2e-api-a-21f543-openshift-default-54699669cc-v57xp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 09 11:47:59.001412 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:47:59.001055    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp" podUID="9683206a-413f-4cb0-a97a-97bbf729bc02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 11:48:00.026069 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:00.026036    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:48:00.046060 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:00.046027    2563 generic.go:356] "Generic (PLEG): container finished" podID="9683206a-413f-4cb0-a97a-97bbf729bc02" containerID="c6775ef44f636ac83b0261c8d3c021f5f2ddd89b661ed2cace17421f749b7a97" exitCode=0
Jul 09 11:48:00.046209 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:00.046105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp" event={"ID":"9683206a-413f-4cb0-a97a-97bbf729bc02","Type":"ContainerDied","Data":"c6775ef44f636ac83b0261c8d3c021f5f2ddd89b661ed2cace17421f749b7a97"}
Jul 09 11:48:00.046456 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:00.046442    2563 scope.go:117] "RemoveContainer" containerID="c6775ef44f636ac83b0261c8d3c021f5f2ddd89b661ed2cace17421f749b7a97"
Jul 09 11:48:00.047355 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:00.047324    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:00.047447 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:00.047405    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:00.047508 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:00.047452    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:01.056520 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:01.056483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp" event={"ID":"9683206a-413f-4cb0-a97a-97bbf729bc02","Type":"ContainerStarted","Data":"5dc2d8559884657cd90dca79750d4561cd693338ac71a888772d4412d6814302"}
Jul 09 11:48:01.497804 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:01.497680    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:48:01.497984 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:01.497837    2563 patch_prober.go:28] interesting pod/e2e-api-a-21f543-openshift-default-54699669cc-v57xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 09 11:48:01.497984 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:01.497887    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp" podUID="9683206a-413f-4cb0-a97a-97bbf729bc02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 09 11:48:02.497092 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:02.497049    2563 patch_prober.go:28] interesting pod/e2e-api-a-21f543-openshift-default-54699669cc-v57xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 09 11:48:02.497513 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:02.497111    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp" podUID="9683206a-413f-4cb0-a97a-97bbf729bc02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 09 11:48:03.496989 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:03.496949    2563 patch_prober.go:28] interesting pod/e2e-api-a-21f543-openshift-default-54699669cc-v57xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 09 11:48:03.497183 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:03.497004    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp" podUID="9683206a-413f-4cb0-a97a-97bbf729bc02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 09 11:48:04.497405 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:04.497358    2563 patch_prober.go:28] interesting pod/e2e-api-a-21f543-openshift-default-54699669cc-v57xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 09 11:48:04.497770 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:04.497443    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp" podUID="9683206a-413f-4cb0-a97a-97bbf729bc02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 09 11:48:04.590360 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:04.590320    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:04.937471 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:04.937438    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:05.247690 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.247597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:05.347577 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.347545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:05.501490 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.501464    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:48:05.501980 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.501891    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:48:05.502871 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.502850    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:48:05.507465 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.507437    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"]
Jul 09 11:48:05.507705 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.507678    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk" podUID="5751ba9d-88ad-4e47-9cbc-3fd7834185e1" containerName="maas-api" containerID="cri-o://e3a1695a2066daa96bd00b22be1466ed991246a3c9f2488d07b2442e04e42002" gracePeriod=30
Jul 09 11:48:05.537243 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.537213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:05.774241 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.774210    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:48:05.801662 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.801634    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-openshift-service-ca\") pod \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\" (UID: \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\") "
Jul 09 11:48:05.801891 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.801695    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-maas-api-tls\") pod \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\" (UID: \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\") "
Jul 09 11:48:05.801891 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.801801    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8knd\" (UniqueName: \"kubernetes.io/projected/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-kube-api-access-h8knd\") pod \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\" (UID: \"5751ba9d-88ad-4e47-9cbc-3fd7834185e1\") "
Jul 09 11:48:05.802097 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.802075    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5751ba9d-88ad-4e47-9cbc-3fd7834185e1" (UID: "5751ba9d-88ad-4e47-9cbc-3fd7834185e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:48:05.804410 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.804380    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5751ba9d-88ad-4e47-9cbc-3fd7834185e1" (UID: "5751ba9d-88ad-4e47-9cbc-3fd7834185e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 11:48:05.804550 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.804416    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-kube-api-access-h8knd" (OuterVolumeSpecName: "kube-api-access-h8knd") pod "5751ba9d-88ad-4e47-9cbc-3fd7834185e1" (UID: "5751ba9d-88ad-4e47-9cbc-3fd7834185e1"). InnerVolumeSpecName "kube-api-access-h8knd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:48:05.903020 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.902986    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:05.903020 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.903016    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8knd\" (UniqueName: \"kubernetes.io/projected/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-kube-api-access-h8knd\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:05.903020 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:05.903026    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5751ba9d-88ad-4e47-9cbc-3fd7834185e1-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:06.087306 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:06.087213    2563 generic.go:356] "Generic (PLEG): container finished" podID="5751ba9d-88ad-4e47-9cbc-3fd7834185e1" containerID="e3a1695a2066daa96bd00b22be1466ed991246a3c9f2488d07b2442e04e42002" exitCode=0
Jul 09 11:48:06.087306 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:06.087258    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"
Jul 09 11:48:06.087524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:06.087298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk" event={"ID":"5751ba9d-88ad-4e47-9cbc-3fd7834185e1","Type":"ContainerDied","Data":"e3a1695a2066daa96bd00b22be1466ed991246a3c9f2488d07b2442e04e42002"}
Jul 09 11:48:06.087524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:06.087364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk" event={"ID":"5751ba9d-88ad-4e47-9cbc-3fd7834185e1","Type":"ContainerDied","Data":"c9bb39a024a4bc69eb107aca99afb065fa96dc0098d49430a9812100fe4b9001"}
Jul 09 11:48:06.087524 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:06.087391    2563 scope.go:117] "RemoveContainer" containerID="e3a1695a2066daa96bd00b22be1466ed991246a3c9f2488d07b2442e04e42002"
Jul 09 11:48:06.096610 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:06.096585    2563 scope.go:117] "RemoveContainer" containerID="e3a1695a2066daa96bd00b22be1466ed991246a3c9f2488d07b2442e04e42002"
Jul 09 11:48:06.096940 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:48:06.096921    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a1695a2066daa96bd00b22be1466ed991246a3c9f2488d07b2442e04e42002\": container with ID starting with e3a1695a2066daa96bd00b22be1466ed991246a3c9f2488d07b2442e04e42002 not found: ID does not exist" containerID="e3a1695a2066daa96bd00b22be1466ed991246a3c9f2488d07b2442e04e42002"
Jul 09 11:48:06.097027 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:06.096952    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a1695a2066daa96bd00b22be1466ed991246a3c9f2488d07b2442e04e42002"} err="failed to get container status \"e3a1695a2066daa96bd00b22be1466ed991246a3c9f2488d07b2442e04e42002\": rpc error: code = NotFound desc = could not find container \"e3a1695a2066daa96bd00b22be1466ed991246a3c9f2488d07b2442e04e42002\": container with ID starting with e3a1695a2066daa96bd00b22be1466ed991246a3c9f2488d07b2442e04e42002 not found: ID does not exist"
Jul 09 11:48:06.142838 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:06.142644    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"]
Jul 09 11:48:06.145905 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:06.145873    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-21f543-864bb74fbb-4n8mk"]
Jul 09 11:48:06.299330 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:06.299291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:06.401595 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:06.401510    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:06.949899 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:06.949868    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5751ba9d-88ad-4e47-9cbc-3fd7834185e1" path="/var/lib/kubelet/pods/5751ba9d-88ad-4e47-9cbc-3fd7834185e1/volumes"
Jul 09 11:48:07.043989 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:07.043940    2563 patch_prober.go:28] interesting pod/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 09 11:48:07.044196 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:07.044010    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks" podUID="092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 11:48:08.099303 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:08.099276    2563 generic.go:356] "Generic (PLEG): container finished" podID="092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" containerID="7891a492c4e9be91469e930ab72fb08ece640b2933f624b66c7a2433c0fe66cc" exitCode=0
Jul 09 11:48:08.099695 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:08.099331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks" event={"ID":"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f","Type":"ContainerDied","Data":"7891a492c4e9be91469e930ab72fb08ece640b2933f624b66c7a2433c0fe66cc"}
Jul 09 11:48:08.099742 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:08.099705    2563 scope.go:117] "RemoveContainer" containerID="7891a492c4e9be91469e930ab72fb08ece640b2933f624b66c7a2433c0fe66cc"
Jul 09 11:48:08.100751 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:08.100721    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:08.100857 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:08.100839    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:08.100898 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:08.100875    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:09.105040 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:09.105003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks" event={"ID":"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f","Type":"ContainerStarted","Data":"7224059d32ef7b799e7336f600a1d42025629c7ddc5a0a7dafbe7acd1a9b2df7"}
Jul 09 11:48:09.527960 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:09.527926    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:48:09.528183 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:09.528161    2563 patch_prober.go:28] interesting pod/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Jul 09 11:48:09.528232 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:09.528210    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks" podUID="092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Jul 09 11:48:10.527690 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:10.527649    2563 patch_prober.go:28] interesting pod/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Jul 09 11:48:10.528117 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:10.527709    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks" podUID="092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Jul 09 11:48:11.052182 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:11.052142    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:11.150989 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:11.150947    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:11.329851 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:11.329749    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"]
Jul 09 11:48:11.330000 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:11.329959    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp" podUID="9683206a-413f-4cb0-a97a-97bbf729bc02" containerName="istio-proxy" containerID="cri-o://5dc2d8559884657cd90dca79750d4561cd693338ac71a888772d4412d6814302" gracePeriod=30
Jul 09 11:48:11.357806 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:11.357750    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:11.527664 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:11.527622    2563 patch_prober.go:28] interesting pod/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Jul 09 11:48:11.527865 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:11.527692    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks" podUID="092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Jul 09 11:48:11.768012 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:11.767979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:12.145182 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:12.145072    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:12.260773 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:12.260709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:12.379115 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:12.379080    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:12.528060 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:12.528023    2563 patch_prober.go:28] interesting pod/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Jul 09 11:48:12.528464 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:12.528087    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks" podUID="092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Jul 09 11:48:12.623646 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:12.623607    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"]
Jul 09 11:48:12.623865 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:12.623840    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch" podUID="1a6a89eb-d786-4f71-a1f0-407fee5c9a51" containerName="maas-api" containerID="cri-o://1e6f045e095396bf55e8dc7bb81f65c22a0822aa48a603a45fab7c7ca78712de" gracePeriod=30
Jul 09 11:48:12.655834 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:12.655754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:12.914093 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:12.914067    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:48:12.974112 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:12.974087    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-openshift-service-ca\") pod \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\" (UID: \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\") "
Jul 09 11:48:12.974295 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:12.974146    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-maas-api-tls\") pod \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\" (UID: \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\") "
Jul 09 11:48:12.974295 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:12.974184    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbffm\" (UniqueName: \"kubernetes.io/projected/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-kube-api-access-kbffm\") pod \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\" (UID: \"1a6a89eb-d786-4f71-a1f0-407fee5c9a51\") "
Jul 09 11:48:12.974486 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:12.974460    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a6a89eb-d786-4f71-a1f0-407fee5c9a51" (UID: "1a6a89eb-d786-4f71-a1f0-407fee5c9a51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:48:12.976258 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:12.976224    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-kube-api-access-kbffm" (OuterVolumeSpecName: "kube-api-access-kbffm") pod "1a6a89eb-d786-4f71-a1f0-407fee5c9a51" (UID: "1a6a89eb-d786-4f71-a1f0-407fee5c9a51"). InnerVolumeSpecName "kube-api-access-kbffm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:48:12.976258 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:12.976249    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a6a89eb-d786-4f71-a1f0-407fee5c9a51" (UID: "1a6a89eb-d786-4f71-a1f0-407fee5c9a51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 11:48:13.075162 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:13.075083    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:13.075162 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:13.075112    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:13.075162 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:13.075122    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbffm\" (UniqueName: \"kubernetes.io/projected/1a6a89eb-d786-4f71-a1f0-407fee5c9a51-kube-api-access-kbffm\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:13.124020 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:13.123986    2563 generic.go:356] "Generic (PLEG): container finished" podID="1a6a89eb-d786-4f71-a1f0-407fee5c9a51" containerID="1e6f045e095396bf55e8dc7bb81f65c22a0822aa48a603a45fab7c7ca78712de" exitCode=0
Jul 09 11:48:13.124210 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:13.124035    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"
Jul 09 11:48:13.124210 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:13.124074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch" event={"ID":"1a6a89eb-d786-4f71-a1f0-407fee5c9a51","Type":"ContainerDied","Data":"1e6f045e095396bf55e8dc7bb81f65c22a0822aa48a603a45fab7c7ca78712de"}
Jul 09 11:48:13.124210 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:13.124126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch" event={"ID":"1a6a89eb-d786-4f71-a1f0-407fee5c9a51","Type":"ContainerDied","Data":"6046873df4b49dfc8b92e7ae5687c6fd455191373e018e7bdb1dc415d4e1f262"}
Jul 09 11:48:13.124210 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:13.124150    2563 scope.go:117] "RemoveContainer" containerID="1e6f045e095396bf55e8dc7bb81f65c22a0822aa48a603a45fab7c7ca78712de"
Jul 09 11:48:13.132680 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:13.132665    2563 scope.go:117] "RemoveContainer" containerID="1e6f045e095396bf55e8dc7bb81f65c22a0822aa48a603a45fab7c7ca78712de"
Jul 09 11:48:13.132954 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:48:13.132935    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6f045e095396bf55e8dc7bb81f65c22a0822aa48a603a45fab7c7ca78712de\": container with ID starting with 1e6f045e095396bf55e8dc7bb81f65c22a0822aa48a603a45fab7c7ca78712de not found: ID does not exist" containerID="1e6f045e095396bf55e8dc7bb81f65c22a0822aa48a603a45fab7c7ca78712de"
Jul 09 11:48:13.133031 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:13.132960    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6f045e095396bf55e8dc7bb81f65c22a0822aa48a603a45fab7c7ca78712de"} err="failed to get container status \"1e6f045e095396bf55e8dc7bb81f65c22a0822aa48a603a45fab7c7ca78712de\": rpc error: code = NotFound desc = could not find container \"1e6f045e095396bf55e8dc7bb81f65c22a0822aa48a603a45fab7c7ca78712de\": container with ID starting with 1e6f045e095396bf55e8dc7bb81f65c22a0822aa48a603a45fab7c7ca78712de not found: ID does not exist"
Jul 09 11:48:13.153734 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:13.153703    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"]
Jul 09 11:48:13.161626 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:13.161601    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9b58c-58df4d757d-cb9ch"]
Jul 09 11:48:13.530870 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:13.530839    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:48:13.531234 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:13.531159    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:48:13.531814 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:13.531774    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:48:14.950435 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:14.950399    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a6a89eb-d786-4f71-a1f0-407fee5c9a51" path="/var/lib/kubelet/pods/1a6a89eb-d786-4f71-a1f0-407fee5c9a51/volumes"
Jul 09 11:48:16.587117 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.587095    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:48:16.605327 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.605246    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9683206a-413f-4cb0-a97a-97bbf729bc02-istiod-ca-cert\") pod \"9683206a-413f-4cb0-a97a-97bbf729bc02\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") "
Jul 09 11:48:16.605327 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.605307    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-token\") pod \"9683206a-413f-4cb0-a97a-97bbf729bc02\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") "
Jul 09 11:48:16.605563 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.605341    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-data\") pod \"9683206a-413f-4cb0-a97a-97bbf729bc02\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") "
Jul 09 11:48:16.605563 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.605374    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-workload-socket\") pod \"9683206a-413f-4cb0-a97a-97bbf729bc02\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") "
Jul 09 11:48:16.605563 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.605431    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-envoy\") pod \"9683206a-413f-4cb0-a97a-97bbf729bc02\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") "
Jul 09 11:48:16.605563 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.605455    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-workload-certs\") pod \"9683206a-413f-4cb0-a97a-97bbf729bc02\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") "
Jul 09 11:48:16.605563 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.605488    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-podinfo\") pod \"9683206a-413f-4cb0-a97a-97bbf729bc02\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") "
Jul 09 11:48:16.605563 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.605503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbbzc\" (UniqueName: \"kubernetes.io/projected/9683206a-413f-4cb0-a97a-97bbf729bc02-kube-api-access-xbbzc\") pod \"9683206a-413f-4cb0-a97a-97bbf729bc02\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") "
Jul 09 11:48:16.605563 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.605539    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-credential-socket\") pod \"9683206a-413f-4cb0-a97a-97bbf729bc02\" (UID: \"9683206a-413f-4cb0-a97a-97bbf729bc02\") "
Jul 09 11:48:16.605946 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.605836    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9683206a-413f-4cb0-a97a-97bbf729bc02" (UID: "9683206a-413f-4cb0-a97a-97bbf729bc02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:48:16.606018 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.605993    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9683206a-413f-4cb0-a97a-97bbf729bc02" (UID: "9683206a-413f-4cb0-a97a-97bbf729bc02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:48:16.606080 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.605652    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9683206a-413f-4cb0-a97a-97bbf729bc02-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9683206a-413f-4cb0-a97a-97bbf729bc02" (UID: "9683206a-413f-4cb0-a97a-97bbf729bc02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:48:16.606660 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.606177    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9683206a-413f-4cb0-a97a-97bbf729bc02" (UID: "9683206a-413f-4cb0-a97a-97bbf729bc02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:48:16.606803 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.606687    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-data" (OuterVolumeSpecName: "istio-data") pod "9683206a-413f-4cb0-a97a-97bbf729bc02" (UID: "9683206a-413f-4cb0-a97a-97bbf729bc02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:48:16.609320 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.608657    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-token" (OuterVolumeSpecName: "istio-token") pod "9683206a-413f-4cb0-a97a-97bbf729bc02" (UID: "9683206a-413f-4cb0-a97a-97bbf729bc02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:48:16.609320 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.608935    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9683206a-413f-4cb0-a97a-97bbf729bc02" (UID: "9683206a-413f-4cb0-a97a-97bbf729bc02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 11:48:16.609581 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.609522    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9683206a-413f-4cb0-a97a-97bbf729bc02-kube-api-access-xbbzc" (OuterVolumeSpecName: "kube-api-access-xbbzc") pod "9683206a-413f-4cb0-a97a-97bbf729bc02" (UID: "9683206a-413f-4cb0-a97a-97bbf729bc02"). InnerVolumeSpecName "kube-api-access-xbbzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:48:16.609809 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.609751    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9683206a-413f-4cb0-a97a-97bbf729bc02" (UID: "9683206a-413f-4cb0-a97a-97bbf729bc02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:48:16.706938 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.706901    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9683206a-413f-4cb0-a97a-97bbf729bc02-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:16.706938 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.706929    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:16.706938 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.706938    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:16.706938 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.706946    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:16.707195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.706955    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:16.707195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.706963    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:16.707195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.706970    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9683206a-413f-4cb0-a97a-97bbf729bc02-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:16.707195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.706978    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbbzc\" (UniqueName: \"kubernetes.io/projected/9683206a-413f-4cb0-a97a-97bbf729bc02-kube-api-access-xbbzc\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:16.707195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:16.706987    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9683206a-413f-4cb0-a97a-97bbf729bc02-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:17.150830 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:17.150769    2563 generic.go:356] "Generic (PLEG): container finished" podID="9683206a-413f-4cb0-a97a-97bbf729bc02" containerID="5dc2d8559884657cd90dca79750d4561cd693338ac71a888772d4412d6814302" exitCode=0
Jul 09 11:48:17.150830 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:17.150813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp" event={"ID":"9683206a-413f-4cb0-a97a-97bbf729bc02","Type":"ContainerDied","Data":"5dc2d8559884657cd90dca79750d4561cd693338ac71a888772d4412d6814302"}
Jul 09 11:48:17.151065 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:17.150840    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"
Jul 09 11:48:17.151065 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:17.150884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp" event={"ID":"9683206a-413f-4cb0-a97a-97bbf729bc02","Type":"ContainerDied","Data":"d95f7e6510905257664114f1f5f5fcd8893bc3b645de5a8f3ddf4ec5d727d676"}
Jul 09 11:48:17.151065 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:17.150915    2563 scope.go:117] "RemoveContainer" containerID="5dc2d8559884657cd90dca79750d4561cd693338ac71a888772d4412d6814302"
Jul 09 11:48:17.159236 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:17.159216    2563 scope.go:117] "RemoveContainer" containerID="c6775ef44f636ac83b0261c8d3c021f5f2ddd89b661ed2cace17421f749b7a97"
Jul 09 11:48:17.166902 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:17.166870    2563 scope.go:117] "RemoveContainer" containerID="5dc2d8559884657cd90dca79750d4561cd693338ac71a888772d4412d6814302"
Jul 09 11:48:17.167243 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:48:17.167222    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc2d8559884657cd90dca79750d4561cd693338ac71a888772d4412d6814302\": container with ID starting with 5dc2d8559884657cd90dca79750d4561cd693338ac71a888772d4412d6814302 not found: ID does not exist" containerID="5dc2d8559884657cd90dca79750d4561cd693338ac71a888772d4412d6814302"
Jul 09 11:48:17.167319 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:17.167251    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc2d8559884657cd90dca79750d4561cd693338ac71a888772d4412d6814302"} err="failed to get container status \"5dc2d8559884657cd90dca79750d4561cd693338ac71a888772d4412d6814302\": rpc error: code = NotFound desc = could not find container \"5dc2d8559884657cd90dca79750d4561cd693338ac71a888772d4412d6814302\": container with ID starting with 5dc2d8559884657cd90dca79750d4561cd693338ac71a888772d4412d6814302 not found: ID does not exist"
Jul 09 11:48:17.167319 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:17.167268    2563 scope.go:117] "RemoveContainer" containerID="c6775ef44f636ac83b0261c8d3c021f5f2ddd89b661ed2cace17421f749b7a97"
Jul 09 11:48:17.167574 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:48:17.167556    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6775ef44f636ac83b0261c8d3c021f5f2ddd89b661ed2cace17421f749b7a97\": container with ID starting with c6775ef44f636ac83b0261c8d3c021f5f2ddd89b661ed2cace17421f749b7a97 not found: ID does not exist" containerID="c6775ef44f636ac83b0261c8d3c021f5f2ddd89b661ed2cace17421f749b7a97"
Jul 09 11:48:17.167631 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:17.167579    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6775ef44f636ac83b0261c8d3c021f5f2ddd89b661ed2cace17421f749b7a97"} err="failed to get container status \"c6775ef44f636ac83b0261c8d3c021f5f2ddd89b661ed2cace17421f749b7a97\": rpc error: code = NotFound desc = could not find container \"c6775ef44f636ac83b0261c8d3c021f5f2ddd89b661ed2cace17421f749b7a97\": container with ID starting with c6775ef44f636ac83b0261c8d3c021f5f2ddd89b661ed2cace17421f749b7a97 not found: ID does not exist"
Jul 09 11:48:17.168586 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:17.168566    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"]
Jul 09 11:48:17.173142 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:17.173122    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-21f543-openshift-default-54699669cc-v57xp"]
Jul 09 11:48:18.335339 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:18.335307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:18.452623 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:18.452584    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:18.617888 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:18.617807    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"]
Jul 09 11:48:18.618038 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:18.617990    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks" podUID="092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" containerName="istio-proxy" containerID="cri-o://7224059d32ef7b799e7336f600a1d42025629c7ddc5a0a7dafbe7acd1a9b2df7" gracePeriod=30
Jul 09 11:48:18.634236 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:18.634202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:18.950406 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:18.950323    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9683206a-413f-4cb0-a97a-97bbf729bc02" path="/var/lib/kubelet/pods/9683206a-413f-4cb0-a97a-97bbf729bc02/volumes"
Jul 09 11:48:19.225179 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.225105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:19.530558 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.530521    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"]
Jul 09 11:48:19.531041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.530832    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5751ba9d-88ad-4e47-9cbc-3fd7834185e1" containerName="maas-api"
Jul 09 11:48:19.531041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.530844    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5751ba9d-88ad-4e47-9cbc-3fd7834185e1" containerName="maas-api"
Jul 09 11:48:19.531041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.530858    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9683206a-413f-4cb0-a97a-97bbf729bc02" containerName="istio-proxy"
Jul 09 11:48:19.531041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.530863    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9683206a-413f-4cb0-a97a-97bbf729bc02" containerName="istio-proxy"
Jul 09 11:48:19.531041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.530876    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a6a89eb-d786-4f71-a1f0-407fee5c9a51" containerName="maas-api"
Jul 09 11:48:19.531041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.530883    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a6a89eb-d786-4f71-a1f0-407fee5c9a51" containerName="maas-api"
Jul 09 11:48:19.531041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.530931    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5751ba9d-88ad-4e47-9cbc-3fd7834185e1" containerName="maas-api"
Jul 09 11:48:19.531041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.530939    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9683206a-413f-4cb0-a97a-97bbf729bc02" containerName="istio-proxy"
Jul 09 11:48:19.531041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.530944    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9683206a-413f-4cb0-a97a-97bbf729bc02" containerName="istio-proxy"
Jul 09 11:48:19.531041 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.530951    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a6a89eb-d786-4f71-a1f0-407fee5c9a51" containerName="maas-api"
Jul 09 11:48:19.533482 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.533465    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.536086 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.536064    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-a3dd9b-openshift-default-dockercfg-hl4sb\""
Jul 09 11:48:19.555618 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.555584    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"]
Jul 09 11:48:19.558288 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.558256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:19.630422 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.630386    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-workload-socket\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.630679 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.630649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-data\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.630837 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.630701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-podinfo\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.630893 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.630837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvt7p\" (UniqueName: \"kubernetes.io/projected/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-kube-api-access-fvt7p\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.630893 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.630883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-envoy\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.631038 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.630924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istiod-ca-cert\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.631038 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.630996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-credential-socket\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.631038 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.631019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-workload-certs\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.631038 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.631033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-token\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.732264 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.732224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-credential-socket\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.732448 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.732279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-workload-certs\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.732448 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.732303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-token\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.732448 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.732341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-workload-socket\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.732448 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.732380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-data\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.732448 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.732444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-podinfo\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.732805 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.732587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvt7p\" (UniqueName: \"kubernetes.io/projected/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-kube-api-access-fvt7p\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.732805 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.732624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-envoy\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.732805 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.732648    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istiod-ca-cert\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.732805 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.732725    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-workload-certs\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.732805 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.732762    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-credential-socket\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.733100 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.733068    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-workload-socket\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.733313 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.733291    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-data\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.733392 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.733292    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istiod-ca-cert\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.735097 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.735068    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-podinfo\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.735185 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.735104    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-envoy\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.742428 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.742410    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-token\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.742605 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.742584    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvt7p\" (UniqueName: \"kubernetes.io/projected/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-kube-api-access-fvt7p\") pod \"e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") " pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.857165 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.857069    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:19.981300 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:19.981251    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:20.000815 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:20.000742    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"]
Jul 09 11:48:20.003083 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:48:20.003056    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968ef205_fbe6_4947_ab53_51a3c4bb0ee6.slice/crio-d6954d8e08580a564344d146388b0d4e220475fe9b713a2c8aafd2070603a58a WatchSource:0}: Error finding container d6954d8e08580a564344d146388b0d4e220475fe9b713a2c8aafd2070603a58a: Status 404 returned error can't find the container with id d6954d8e08580a564344d146388b0d4e220475fe9b713a2c8aafd2070603a58a
Jul 09 11:48:20.005066 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:20.005039    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:20.005125 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:20.005101    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:20.005166 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:20.005129    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:20.168763 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:20.168678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w" event={"ID":"968ef205-fbe6-4947-ab53-51a3c4bb0ee6","Type":"ContainerStarted","Data":"e87de1858bc28d4fdae7db93b0aae04ba28ac7edcc5fdfe6658c1ebc1853e24b"}
Jul 09 11:48:20.168763 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:20.168740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w" event={"ID":"968ef205-fbe6-4947-ab53-51a3c4bb0ee6","Type":"ContainerStarted","Data":"d6954d8e08580a564344d146388b0d4e220475fe9b713a2c8aafd2070603a58a"}
Jul 09 11:48:20.195229 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:20.195168    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w" podStartSLOduration=1.195148882 podStartE2EDuration="1.195148882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:48:20.193939407 +0000 UTC m=+2183.785670572" watchObservedRunningTime="2026-07-09 11:48:20.195148882 +0000 UTC m=+2183.786880072"
Jul 09 11:48:20.857527 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:20.857482    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:21.862018 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:21.861984    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:22.178398 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.178324    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:22.179416 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.179393    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:22.224292 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.224259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:22.302063 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.302026    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"]
Jul 09 11:48:22.302370 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.302350    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9683206a-413f-4cb0-a97a-97bbf729bc02" containerName="istio-proxy"
Jul 09 11:48:22.302370 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.302369    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9683206a-413f-4cb0-a97a-97bbf729bc02" containerName="istio-proxy"
Jul 09 11:48:22.304370 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.304352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:48:22.306736 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.306720    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a3dd9b-serving-cert\""
Jul 09 11:48:22.308327 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.308305    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:22.318265 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.318241    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"]
Jul 09 11:48:22.354926 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.354899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42bcz\" (UniqueName: \"kubernetes.io/projected/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-kube-api-access-42bcz\") pod \"maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg\" (UID: \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:48:22.355081 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.354958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-maas-api-tls\") pod \"maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg\" (UID: \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:48:22.355081 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.354986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg\" (UID: \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:48:22.395273 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.395233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:22.455731 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.455640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-maas-api-tls\") pod \"maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg\" (UID: \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:48:22.455731 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.455676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg\" (UID: \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:48:22.455731 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.455733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42bcz\" (UniqueName: \"kubernetes.io/projected/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-kube-api-access-42bcz\") pod \"maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg\" (UID: \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:48:22.455995 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:48:22.455832    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-serving-cert: secret "maas-api-e2e-shared-a-a3dd9b-serving-cert" not found
Jul 09 11:48:22.455995 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:48:22.455910    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-maas-api-tls podName:0f9f94b7-0542-4a24-952b-c63ce6bb2eef nodeName:}" failed. No retries permitted until 2026-07-09 11:48:22.955887455 +0000 UTC m=+2186.547618616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-maas-api-tls") pod "maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg" (UID: "0f9f94b7-0542-4a24-952b-c63ce6bb2eef") : secret "maas-api-e2e-shared-a-a3dd9b-serving-cert" not found
Jul 09 11:48:22.456387 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.456368    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg\" (UID: \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:48:22.466840 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.466816    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42bcz\" (UniqueName: \"kubernetes.io/projected/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-kube-api-access-42bcz\") pod \"maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg\" (UID: \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:48:22.892661 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.892617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:22.960377 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:22.960345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-maas-api-tls\") pod \"maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg\" (UID: \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:48:22.960547 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:48:22.960501    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-serving-cert: secret "maas-api-e2e-shared-a-a3dd9b-serving-cert" not found
Jul 09 11:48:22.960607 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:48:22.960566    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-maas-api-tls podName:0f9f94b7-0542-4a24-952b-c63ce6bb2eef nodeName:}" failed. No retries permitted until 2026-07-09 11:48:23.960550794 +0000 UTC m=+2187.552281940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-maas-api-tls") pod "maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg" (UID: "0f9f94b7-0542-4a24-952b-c63ce6bb2eef") : secret "maas-api-e2e-shared-a-a3dd9b-serving-cert" not found
Jul 09 11:48:23.014964 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.014924    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:23.115661 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.115626    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:23.868906 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.868881    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:48:23.970132 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.970034    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-envoy\") pod \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") "
Jul 09 11:48:23.970132 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.970083    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-credential-socket\") pod \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") "
Jul 09 11:48:23.970132 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.970117    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-data\") pod \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") "
Jul 09 11:48:23.970697 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.970139    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istiod-ca-cert\") pod \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") "
Jul 09 11:48:23.970697 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.970231    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-token\") pod \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") "
Jul 09 11:48:23.970697 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.970267    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-workload-certs\") pod \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") "
Jul 09 11:48:23.970697 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.970312    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-podinfo\") pod \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") "
Jul 09 11:48:23.970697 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.970340    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-workload-socket\") pod \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") "
Jul 09 11:48:23.970697 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.970376    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nwp8\" (UniqueName: \"kubernetes.io/projected/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-kube-api-access-8nwp8\") pod \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\" (UID: \"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f\") "
Jul 09 11:48:23.970697 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.970481    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" (UID: "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:48:23.970697 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.970646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-maas-api-tls\") pod \"maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg\" (UID: \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:48:23.970697 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.970674    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" (UID: "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:48:23.971195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.970756    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:23.971195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.970777    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:23.971195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.970889    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" (UID: "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:48:23.971195 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.971098    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" (UID: "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:48:23.971371 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.971251    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-data" (OuterVolumeSpecName: "istio-data") pod "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" (UID: "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:48:23.972657 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.972636    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" (UID: "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 11:48:23.972896 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.972876    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" (UID: "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:48:23.972981 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.972908    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-kube-api-access-8nwp8" (OuterVolumeSpecName: "kube-api-access-8nwp8") pod "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" (UID: "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f"). InnerVolumeSpecName "kube-api-access-8nwp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:48:23.973036 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.973010    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" (UID: "092d17d8-c2d9-4938-8b71-c70f5e5c0a2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:48:23.973364 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:23.973348    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-maas-api-tls\") pod \"maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg\" (UID: \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:48:24.071480 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.071444    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:24.071480 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.071472    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:24.071480 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.071482    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:24.071480 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.071490    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:24.071777 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.071501    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nwp8\" (UniqueName: \"kubernetes.io/projected/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-kube-api-access-8nwp8\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:24.071777 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.071511    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:24.071777 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.071518    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:48:24.115377 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.115343    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:48:24.188974 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.188939    2563 generic.go:356] "Generic (PLEG): container finished" podID="092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" containerID="7224059d32ef7b799e7336f600a1d42025629c7ddc5a0a7dafbe7acd1a9b2df7" exitCode=0
Jul 09 11:48:24.189127 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.189001    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"
Jul 09 11:48:24.189127 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.189018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks" event={"ID":"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f","Type":"ContainerDied","Data":"7224059d32ef7b799e7336f600a1d42025629c7ddc5a0a7dafbe7acd1a9b2df7"}
Jul 09 11:48:24.189127 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.189085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks" event={"ID":"092d17d8-c2d9-4938-8b71-c70f5e5c0a2f","Type":"ContainerDied","Data":"7ebc001444c88f5d6da1fd94a7f8e7e5bdbbb78c184f2981fffbbc397db84475"}
Jul 09 11:48:24.189127 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.189115    2563 scope.go:117] "RemoveContainer" containerID="7224059d32ef7b799e7336f600a1d42025629c7ddc5a0a7dafbe7acd1a9b2df7"
Jul 09 11:48:24.198846 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.198803    2563 scope.go:117] "RemoveContainer" containerID="7891a492c4e9be91469e930ab72fb08ece640b2933f624b66c7a2433c0fe66cc"
Jul 09 11:48:24.209921 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.209756    2563 scope.go:117] "RemoveContainer" containerID="7224059d32ef7b799e7336f600a1d42025629c7ddc5a0a7dafbe7acd1a9b2df7"
Jul 09 11:48:24.210310 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:48:24.210284    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7224059d32ef7b799e7336f600a1d42025629c7ddc5a0a7dafbe7acd1a9b2df7\": container with ID starting with 7224059d32ef7b799e7336f600a1d42025629c7ddc5a0a7dafbe7acd1a9b2df7 not found: ID does not exist" containerID="7224059d32ef7b799e7336f600a1d42025629c7ddc5a0a7dafbe7acd1a9b2df7"
Jul 09 11:48:24.210418 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.210324    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7224059d32ef7b799e7336f600a1d42025629c7ddc5a0a7dafbe7acd1a9b2df7"} err="failed to get container status \"7224059d32ef7b799e7336f600a1d42025629c7ddc5a0a7dafbe7acd1a9b2df7\": rpc error: code = NotFound desc = could not find container \"7224059d32ef7b799e7336f600a1d42025629c7ddc5a0a7dafbe7acd1a9b2df7\": container with ID starting with 7224059d32ef7b799e7336f600a1d42025629c7ddc5a0a7dafbe7acd1a9b2df7 not found: ID does not exist"
Jul 09 11:48:24.210418 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.210350    2563 scope.go:117] "RemoveContainer" containerID="7891a492c4e9be91469e930ab72fb08ece640b2933f624b66c7a2433c0fe66cc"
Jul 09 11:48:24.210719 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:48:24.210697    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7891a492c4e9be91469e930ab72fb08ece640b2933f624b66c7a2433c0fe66cc\": container with ID starting with 7891a492c4e9be91469e930ab72fb08ece640b2933f624b66c7a2433c0fe66cc not found: ID does not exist" containerID="7891a492c4e9be91469e930ab72fb08ece640b2933f624b66c7a2433c0fe66cc"
Jul 09 11:48:24.210815 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.210726    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7891a492c4e9be91469e930ab72fb08ece640b2933f624b66c7a2433c0fe66cc"} err="failed to get container status \"7891a492c4e9be91469e930ab72fb08ece640b2933f624b66c7a2433c0fe66cc\": rpc error: code = NotFound desc = could not find container \"7891a492c4e9be91469e930ab72fb08ece640b2933f624b66c7a2433c0fe66cc\": container with ID starting with 7891a492c4e9be91469e930ab72fb08ece640b2933f624b66c7a2433c0fe66cc not found: ID does not exist"
Jul 09 11:48:24.216394 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.216374    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"]
Jul 09 11:48:24.221060 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.220999    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e9b58c-openshift-default-66db7c968f-mnfks"]
Jul 09 11:48:24.257454 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.257428    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"]
Jul 09 11:48:24.259735 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:48:24.259705    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9f94b7_0542_4a24_952b_c63ce6bb2eef.slice/crio-c536ac556e17d31a992996e513ee89add3a25ca7515d24fc1368d3cc001adaa7 WatchSource:0}: Error finding container c536ac556e17d31a992996e513ee89add3a25ca7515d24fc1368d3cc001adaa7: Status 404 returned error can't find the container with id c536ac556e17d31a992996e513ee89add3a25ca7515d24fc1368d3cc001adaa7
Jul 09 11:48:24.951144 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:24.951050    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" path="/var/lib/kubelet/pods/092d17d8-c2d9-4938-8b71-c70f5e5c0a2f/volumes"
Jul 09 11:48:25.195984 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:25.195951    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg" event={"ID":"0f9f94b7-0542-4a24-952b-c63ce6bb2eef","Type":"ContainerStarted","Data":"78a1d2fa063287c45cdda4a7577c3be74c96d100d8011101b845d59de870ff56"}
Jul 09 11:48:25.196472 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:25.196005    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg" event={"ID":"0f9f94b7-0542-4a24-952b-c63ce6bb2eef","Type":"ContainerStarted","Data":"c536ac556e17d31a992996e513ee89add3a25ca7515d24fc1368d3cc001adaa7"}
Jul 09 11:48:25.196472 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:25.196058    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:48:25.215291 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:25.215193    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg" podStartSLOduration=2.821260605 podStartE2EDuration="3.215177923s" totalImagesPullingTime="393.917318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:48:25.213366607 +0000 UTC m=+2188.805097774" watchObservedRunningTime="2026-07-09 11:48:25.215177923 +0000 UTC m=+2188.806909086"
Jul 09 11:48:26.730518 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:26.730469    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:27.059304 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.059267    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:27.334713 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.334632    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"]
Jul 09 11:48:27.335196 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.335167    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" containerName="istio-proxy"
Jul 09 11:48:27.335196 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.335187    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" containerName="istio-proxy"
Jul 09 11:48:27.335357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.335216    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" containerName="istio-proxy"
Jul 09 11:48:27.335357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.335225    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" containerName="istio-proxy"
Jul 09 11:48:27.335357 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.335301    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" containerName="istio-proxy"
Jul 09 11:48:27.338853 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.338835    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.344914 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.344891    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-02d29c-openshift-default-dockercfg-hszd4\""
Jul 09 11:48:27.349332 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.349306    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"]
Jul 09 11:48:27.362597 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.362566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:27.401544 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.401508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-workload-socket\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.401544 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.401540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-istio-envoy\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.401755 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.401609    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58mr2\" (UniqueName: \"kubernetes.io/projected/d6cece14-7622-4e7d-a754-9b606afba55f-kube-api-access-58mr2\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.401755 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.401647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6cece14-7622-4e7d-a754-9b606afba55f-istiod-ca-cert\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.401755 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.401665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-credential-socket\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.401755 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.401682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6cece14-7622-4e7d-a754-9b606afba55f-istio-podinfo\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.401755 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.401708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-istio-data\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.401755 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.401722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6cece14-7622-4e7d-a754-9b606afba55f-istio-token\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.401755 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.401740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-workload-certs\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.502969 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.502938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-istio-envoy\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.503254 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.502996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58mr2\" (UniqueName: \"kubernetes.io/projected/d6cece14-7622-4e7d-a754-9b606afba55f-kube-api-access-58mr2\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.503254 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.503023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6cece14-7622-4e7d-a754-9b606afba55f-istiod-ca-cert\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.503254 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.503041    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-credential-socket\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.503254 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.503058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6cece14-7622-4e7d-a754-9b606afba55f-istio-podinfo\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.503254 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.503085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-istio-data\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.503254 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.503100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6cece14-7622-4e7d-a754-9b606afba55f-istio-token\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.503254 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.503124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-workload-certs\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.503254 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.503198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-workload-socket\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.503692 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.503436    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-credential-socket\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.503692 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.503470    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-workload-socket\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.503797 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.503691    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-workload-certs\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.503797 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.503759    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-istio-data\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.503918 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.503857    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6cece14-7622-4e7d-a754-9b606afba55f-istiod-ca-cert\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.505248 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.505228    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-istio-envoy\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.505754 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.505736    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6cece14-7622-4e7d-a754-9b606afba55f-istio-podinfo\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.511773 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.511743    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6cece14-7622-4e7d-a754-9b606afba55f-istio-token\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.511907 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.511889    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58mr2\" (UniqueName: \"kubernetes.io/projected/d6cece14-7622-4e7d-a754-9b606afba55f-kube-api-access-58mr2\") pod \"e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") " pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.649821 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.649712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:27.800815 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.800652    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"]
Jul 09 11:48:27.806354 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.806307    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:27.806448 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.806398    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:27.806448 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.806436    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:27.810002 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:27.809979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:28.212355 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:28.212318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq" event={"ID":"d6cece14-7622-4e7d-a754-9b606afba55f","Type":"ContainerStarted","Data":"c364eda291fd41f9fec780c141f676f3d443ecb9a5a8274377ab1ba56c2fef08"}
Jul 09 11:48:28.212563 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:28.212375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq" event={"ID":"d6cece14-7622-4e7d-a754-9b606afba55f","Type":"ContainerStarted","Data":"2b2f8dd9649e85bc358475a4ae1edf8f2e9717e06736b83c8a8621d5bd19cc7f"}
Jul 09 11:48:28.235372 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:28.235306    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq" podStartSLOduration=1.235283871 podStartE2EDuration="1.235283871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:48:28.232544768 +0000 UTC m=+2191.824275932" watchObservedRunningTime="2026-07-09 11:48:28.235283871 +0000 UTC m=+2191.827015037"
Jul 09 11:48:28.650509 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:28.650468    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:28.663344 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:28.663304    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:29.217663 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:29.217634    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:29.218511 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:29.218492    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:30.345819 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.345765    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:30.478915 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.478877    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"]
Jul 09 11:48:30.479448 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.479430    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="092d17d8-c2d9-4938-8b71-c70f5e5c0a2f" containerName="istio-proxy"
Jul 09 11:48:30.489836 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.489806    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"
Jul 09 11:48:30.494385 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.494360    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-02d29c-serving-cert\""
Jul 09 11:48:30.494951 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.494929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:30.501306 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.501284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"]
Jul 09 11:48:30.588334 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.588291    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:30.630547 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.630461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/828a337a-7887-4ea6-af95-ef2363365162-maas-api-tls\") pod \"maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w\" (UID: \"828a337a-7887-4ea6-af95-ef2363365162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"
Jul 09 11:48:30.630547 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.630504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t4hc\" (UniqueName: \"kubernetes.io/projected/828a337a-7887-4ea6-af95-ef2363365162-kube-api-access-8t4hc\") pod \"maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w\" (UID: \"828a337a-7887-4ea6-af95-ef2363365162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"
Jul 09 11:48:30.630547 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.630526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/828a337a-7887-4ea6-af95-ef2363365162-openshift-service-ca\") pod \"maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w\" (UID: \"828a337a-7887-4ea6-af95-ef2363365162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"
Jul 09 11:48:30.706573 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.706537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:30.731955 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.731915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/828a337a-7887-4ea6-af95-ef2363365162-maas-api-tls\") pod \"maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w\" (UID: \"828a337a-7887-4ea6-af95-ef2363365162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"
Jul 09 11:48:30.732093 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.731980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t4hc\" (UniqueName: \"kubernetes.io/projected/828a337a-7887-4ea6-af95-ef2363365162-kube-api-access-8t4hc\") pod \"maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w\" (UID: \"828a337a-7887-4ea6-af95-ef2363365162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"
Jul 09 11:48:30.732093 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.732002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/828a337a-7887-4ea6-af95-ef2363365162-openshift-service-ca\") pod \"maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w\" (UID: \"828a337a-7887-4ea6-af95-ef2363365162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"
Jul 09 11:48:30.732692 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.732670    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/828a337a-7887-4ea6-af95-ef2363365162-openshift-service-ca\") pod \"maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w\" (UID: \"828a337a-7887-4ea6-af95-ef2363365162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"
Jul 09 11:48:30.734374 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.734353    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/828a337a-7887-4ea6-af95-ef2363365162-maas-api-tls\") pod \"maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w\" (UID: \"828a337a-7887-4ea6-af95-ef2363365162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"
Jul 09 11:48:30.749159 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.749093    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t4hc\" (UniqueName: \"kubernetes.io/projected/828a337a-7887-4ea6-af95-ef2363365162-kube-api-access-8t4hc\") pod \"maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w\" (UID: \"828a337a-7887-4ea6-af95-ef2363365162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"
Jul 09 11:48:30.801595 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.801559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"
Jul 09 11:48:30.974264 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:30.974133    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"]
Jul 09 11:48:30.977216 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:48:30.977043    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828a337a_7887_4ea6_af95_ef2363365162.slice/crio-ba90cf6ac6e2a16ccecba1cbf4eb5f21975ce9595582270932a1ef3f36dcd767 WatchSource:0}: Error finding container ba90cf6ac6e2a16ccecba1cbf4eb5f21975ce9595582270932a1ef3f36dcd767: Status 404 returned error can't find the container with id ba90cf6ac6e2a16ccecba1cbf4eb5f21975ce9595582270932a1ef3f36dcd767
Jul 09 11:48:31.037305 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:31.037276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:31.142021 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:31.141939    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:31.207582 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:31.207363    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:48:31.227859 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:31.227764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w" event={"ID":"828a337a-7887-4ea6-af95-ef2363365162","Type":"ContainerStarted","Data":"ba90cf6ac6e2a16ccecba1cbf4eb5f21975ce9595582270932a1ef3f36dcd767"}
Jul 09 11:48:32.234071 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:32.234024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w" event={"ID":"828a337a-7887-4ea6-af95-ef2363365162","Type":"ContainerStarted","Data":"d0462aa09ee2dffbe1925978b6ae9047cda9dbb57140e6515e99230f9fc6418e"}
Jul 09 11:48:32.234531 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:32.234201    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"
Jul 09 11:48:32.252929 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:32.252860    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w" podStartSLOduration=1.930143949 podStartE2EDuration="2.252836762s" totalImagesPullingTime="322.692813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 11:48:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:48:32.250301127 +0000 UTC m=+2195.842032292" watchObservedRunningTime="2026-07-09 11:48:32.252836762 +0000 UTC m=+2195.844567929"
Jul 09 11:48:34.735845 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:34.735813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:35.101917 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:35.101873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:35.251368 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:35.251340    2563 generic.go:356] "Generic (PLEG): container finished" podID="968ef205-fbe6-4947-ab53-51a3c4bb0ee6" containerID="e87de1858bc28d4fdae7db93b0aae04ba28ac7edcc5fdfe6658c1ebc1853e24b" exitCode=0
Jul 09 11:48:35.251527 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:35.251414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w" event={"ID":"968ef205-fbe6-4947-ab53-51a3c4bb0ee6","Type":"ContainerDied","Data":"e87de1858bc28d4fdae7db93b0aae04ba28ac7edcc5fdfe6658c1ebc1853e24b"}
Jul 09 11:48:35.251832 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:35.251819    2563 scope.go:117] "RemoveContainer" containerID="e87de1858bc28d4fdae7db93b0aae04ba28ac7edcc5fdfe6658c1ebc1853e24b"
Jul 09 11:48:35.252802 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:35.252756    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:35.252930 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:35.252844    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:35.252930 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:35.252878    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:36.261635 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:36.261575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w" event={"ID":"968ef205-fbe6-4947-ab53-51a3c4bb0ee6","Type":"ContainerStarted","Data":"c3c6eb2ff88202d61979f0533513d00d65d4fe5a3712a1d597cf3aa87ffe3e23"}
Jul 09 11:48:36.857933 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:36.857897    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:36.858933 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:36.858901    2563 patch_prober.go:28] interesting pod/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Jul 09 11:48:36.859163 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:36.859136    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w" podUID="968ef205-fbe6-4947-ab53-51a3c4bb0ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 09 11:48:37.178582 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:37.178495    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:37.858479 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:37.858427    2563 patch_prober.go:28] interesting pod/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Jul 09 11:48:37.859023 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:37.858503    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w" podUID="968ef205-fbe6-4947-ab53-51a3c4bb0ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 09 11:48:38.255193 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:38.255158    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"
Jul 09 11:48:38.858510 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:38.858070    2563 patch_prober.go:28] interesting pod/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Jul 09 11:48:38.858510 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:38.858138    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w" podUID="968ef205-fbe6-4947-ab53-51a3c4bb0ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 09 11:48:39.857523 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:39.857482    2563 patch_prober.go:28] interesting pod/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Jul 09 11:48:39.858038 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:39.858008    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w" podUID="968ef205-fbe6-4947-ab53-51a3c4bb0ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Jul 09 11:48:40.864473 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:40.864439    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:40.865521 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:40.865499    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:48:43.628913 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:43.628869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:43.743168 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:43.743133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:44.219561 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:44.219532    2563 patch_prober.go:28] interesting pod/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Jul 09 11:48:44.219717 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:44.219590    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq" podUID="d6cece14-7622-4e7d-a754-9b606afba55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Jul 09 11:48:44.306993 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:44.306963    2563 generic.go:356] "Generic (PLEG): container finished" podID="d6cece14-7622-4e7d-a754-9b606afba55f" containerID="c364eda291fd41f9fec780c141f676f3d443ecb9a5a8274377ab1ba56c2fef08" exitCode=0
Jul 09 11:48:44.307199 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:44.307030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq" event={"ID":"d6cece14-7622-4e7d-a754-9b606afba55f","Type":"ContainerDied","Data":"c364eda291fd41f9fec780c141f676f3d443ecb9a5a8274377ab1ba56c2fef08"}
Jul 09 11:48:44.307443 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:44.307427    2563 scope.go:117] "RemoveContainer" containerID="c364eda291fd41f9fec780c141f676f3d443ecb9a5a8274377ab1ba56c2fef08"
Jul 09 11:48:44.308456 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:44.308420    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:44.308533 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:44.308516    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:44.308578 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:44.308565    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 09 11:48:45.314034 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:45.313998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq" event={"ID":"d6cece14-7622-4e7d-a754-9b606afba55f","Type":"ContainerStarted","Data":"fbb53f68f00635763f694e27f02f5d67fb3574afc919545d8375b2e2d3a3fb0f"}
Jul 09 11:48:45.650680 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:45.650559    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:45.650885 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:45.650708    2563 patch_prober.go:28] interesting pod/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Jul 09 11:48:45.650885 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:45.650764    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq" podUID="d6cece14-7622-4e7d-a754-9b606afba55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Jul 09 11:48:46.650577 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:46.650525    2563 patch_prober.go:28] interesting pod/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Jul 09 11:48:46.651157 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:46.650599    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq" podUID="d6cece14-7622-4e7d-a754-9b606afba55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Jul 09 11:48:47.650186 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:47.650148    2563 patch_prober.go:28] interesting pod/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Jul 09 11:48:47.650500 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:47.650467    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq" podUID="d6cece14-7622-4e7d-a754-9b606afba55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Jul 09 11:48:48.650633 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:48.650590    2563 patch_prober.go:28] interesting pod/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Jul 09 11:48:48.651278 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:48.651245    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq" podUID="d6cece14-7622-4e7d-a754-9b606afba55f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Jul 09 11:48:49.654582 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:49.654555    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:49.654968 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:49.654879    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:49.655443 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:49.655427    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:48:52.009688 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:52.009603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:48:52.102635 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:48:52.102601    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:49:49.299405 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:49:49.299319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:49:53.304848 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:49:53.304806    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:49:53.401384 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:49:53.401347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:50:09.969511 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:50:09.969474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:50:14.281080 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:50:14.281039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:40.025382 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:40.025340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:40.618800 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:40.618734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:40.689755 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:40.689714    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"]
Jul 09 11:51:40.689987 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:40.689958    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg" podUID="0f9f94b7-0542-4a24-952b-c63ce6bb2eef" containerName="maas-api" containerID="cri-o://78a1d2fa063287c45cdda4a7577c3be74c96d100d8011101b845d59de870ff56" gracePeriod=30
Jul 09 11:51:40.693889 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:40.693851    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:40.755265 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:40.755231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:40.927743 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:40.927719    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:51:41.035013 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.034970    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-openshift-service-ca\") pod \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\" (UID: \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\") "
Jul 09 11:51:41.035422 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.035066    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-maas-api-tls\") pod \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\" (UID: \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\") "
Jul 09 11:51:41.035422 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.035117    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42bcz\" (UniqueName: \"kubernetes.io/projected/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-kube-api-access-42bcz\") pod \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\" (UID: \"0f9f94b7-0542-4a24-952b-c63ce6bb2eef\") "
Jul 09 11:51:41.035422 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.035362    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f9f94b7-0542-4a24-952b-c63ce6bb2eef" (UID: "0f9f94b7-0542-4a24-952b-c63ce6bb2eef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:51:41.037287 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.037250    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f9f94b7-0542-4a24-952b-c63ce6bb2eef" (UID: "0f9f94b7-0542-4a24-952b-c63ce6bb2eef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 11:51:41.037414 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.037324    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-kube-api-access-42bcz" (OuterVolumeSpecName: "kube-api-access-42bcz") pod "0f9f94b7-0542-4a24-952b-c63ce6bb2eef" (UID: "0f9f94b7-0542-4a24-952b-c63ce6bb2eef"). InnerVolumeSpecName "kube-api-access-42bcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:51:41.128133 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.128100    2563 generic.go:356] "Generic (PLEG): container finished" podID="0f9f94b7-0542-4a24-952b-c63ce6bb2eef" containerID="78a1d2fa063287c45cdda4a7577c3be74c96d100d8011101b845d59de870ff56" exitCode=0
Jul 09 11:51:41.128304 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.128140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg" event={"ID":"0f9f94b7-0542-4a24-952b-c63ce6bb2eef","Type":"ContainerDied","Data":"78a1d2fa063287c45cdda4a7577c3be74c96d100d8011101b845d59de870ff56"}
Jul 09 11:51:41.128304 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.128149    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"
Jul 09 11:51:41.128304 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.128180    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg" event={"ID":"0f9f94b7-0542-4a24-952b-c63ce6bb2eef","Type":"ContainerDied","Data":"c536ac556e17d31a992996e513ee89add3a25ca7515d24fc1368d3cc001adaa7"}
Jul 09 11:51:41.128304 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.128203    2563 scope.go:117] "RemoveContainer" containerID="78a1d2fa063287c45cdda4a7577c3be74c96d100d8011101b845d59de870ff56"
Jul 09 11:51:41.136020 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.135999    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:41.136020 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.136020    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42bcz\" (UniqueName: \"kubernetes.io/projected/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-kube-api-access-42bcz\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:41.136184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.136029    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f9f94b7-0542-4a24-952b-c63ce6bb2eef-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:41.137246 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.137230    2563 scope.go:117] "RemoveContainer" containerID="78a1d2fa063287c45cdda4a7577c3be74c96d100d8011101b845d59de870ff56"
Jul 09 11:51:41.137457 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:51:41.137439    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a1d2fa063287c45cdda4a7577c3be74c96d100d8011101b845d59de870ff56\": container with ID starting with 78a1d2fa063287c45cdda4a7577c3be74c96d100d8011101b845d59de870ff56 not found: ID does not exist" containerID="78a1d2fa063287c45cdda4a7577c3be74c96d100d8011101b845d59de870ff56"
Jul 09 11:51:41.137509 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.137463    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a1d2fa063287c45cdda4a7577c3be74c96d100d8011101b845d59de870ff56"} err="failed to get container status \"78a1d2fa063287c45cdda4a7577c3be74c96d100d8011101b845d59de870ff56\": rpc error: code = NotFound desc = could not find container \"78a1d2fa063287c45cdda4a7577c3be74c96d100d8011101b845d59de870ff56\": container with ID starting with 78a1d2fa063287c45cdda4a7577c3be74c96d100d8011101b845d59de870ff56 not found: ID does not exist"
Jul 09 11:51:41.153387 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.153363    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"]
Jul 09 11:51:41.156984 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:41.156956    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a3dd9b-7f4d448949-z7lgg"]
Jul 09 11:51:42.950203 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:42.950165    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f9f94b7-0542-4a24-952b-c63ce6bb2eef" path="/var/lib/kubelet/pods/0f9f94b7-0542-4a24-952b-c63ce6bb2eef/volumes"
Jul 09 11:51:46.463050 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:46.463008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:46.554328 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:46.554296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:46.726300 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:46.726215    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"]
Jul 09 11:51:46.726482 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:46.726441    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w" podUID="968ef205-fbe6-4947-ab53-51a3c4bb0ee6" containerName="istio-proxy" containerID="cri-o://c3c6eb2ff88202d61979f0533513d00d65d4fe5a3712a1d597cf3aa87ffe3e23" gracePeriod=30
Jul 09 11:51:46.770673 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:46.770640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:47.211929 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:47.211896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:47.601043 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:47.601000    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:47.701825 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:47.701777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:47.804514 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:47.804456    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:47.921526 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:47.921442    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"]
Jul 09 11:51:47.921719 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:47.921693    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w" podUID="828a337a-7887-4ea6-af95-ef2363365162" containerName="maas-api" containerID="cri-o://d0462aa09ee2dffbe1925978b6ae9047cda9dbb57140e6515e99230f9fc6418e" gracePeriod=30
Jul 09 11:51:47.943409 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:47.943372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:48.162132 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:48.162102    2563 generic.go:356] "Generic (PLEG): container finished" podID="828a337a-7887-4ea6-af95-ef2363365162" containerID="d0462aa09ee2dffbe1925978b6ae9047cda9dbb57140e6515e99230f9fc6418e" exitCode=0
Jul 09 11:51:48.162268 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:48.162138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w" event={"ID":"828a337a-7887-4ea6-af95-ef2363365162","Type":"ContainerDied","Data":"d0462aa09ee2dffbe1925978b6ae9047cda9dbb57140e6515e99230f9fc6418e"}
Jul 09 11:51:48.162268 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:48.162180    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w" event={"ID":"828a337a-7887-4ea6-af95-ef2363365162","Type":"ContainerDied","Data":"ba90cf6ac6e2a16ccecba1cbf4eb5f21975ce9595582270932a1ef3f36dcd767"}
Jul 09 11:51:48.162268 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:48.162197    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba90cf6ac6e2a16ccecba1cbf4eb5f21975ce9595582270932a1ef3f36dcd767"
Jul 09 11:51:48.164968 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:48.164942    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"
Jul 09 11:51:48.298484 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:48.298452    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t4hc\" (UniqueName: \"kubernetes.io/projected/828a337a-7887-4ea6-af95-ef2363365162-kube-api-access-8t4hc\") pod \"828a337a-7887-4ea6-af95-ef2363365162\" (UID: \"828a337a-7887-4ea6-af95-ef2363365162\") "
Jul 09 11:51:48.298657 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:48.298496    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/828a337a-7887-4ea6-af95-ef2363365162-maas-api-tls\") pod \"828a337a-7887-4ea6-af95-ef2363365162\" (UID: \"828a337a-7887-4ea6-af95-ef2363365162\") "
Jul 09 11:51:48.298657 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:48.298621    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/828a337a-7887-4ea6-af95-ef2363365162-openshift-service-ca\") pod \"828a337a-7887-4ea6-af95-ef2363365162\" (UID: \"828a337a-7887-4ea6-af95-ef2363365162\") "
Jul 09 11:51:48.298992 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:48.298966    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/828a337a-7887-4ea6-af95-ef2363365162-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "828a337a-7887-4ea6-af95-ef2363365162" (UID: "828a337a-7887-4ea6-af95-ef2363365162"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:51:48.300621 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:48.300603    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828a337a-7887-4ea6-af95-ef2363365162-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "828a337a-7887-4ea6-af95-ef2363365162" (UID: "828a337a-7887-4ea6-af95-ef2363365162"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 11:51:48.300741 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:48.300723    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828a337a-7887-4ea6-af95-ef2363365162-kube-api-access-8t4hc" (OuterVolumeSpecName: "kube-api-access-8t4hc") pod "828a337a-7887-4ea6-af95-ef2363365162" (UID: "828a337a-7887-4ea6-af95-ef2363365162"). InnerVolumeSpecName "kube-api-access-8t4hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:51:48.399347 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:48.399301    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t4hc\" (UniqueName: \"kubernetes.io/projected/828a337a-7887-4ea6-af95-ef2363365162-kube-api-access-8t4hc\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:48.399347 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:48.399338    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/828a337a-7887-4ea6-af95-ef2363365162-maas-api-tls\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:48.399347 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:48.399354    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/828a337a-7887-4ea6-af95-ef2363365162-openshift-service-ca\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:49.166565 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:49.166537    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"
Jul 09 11:51:49.184916 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:49.184889    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"]
Jul 09 11:51:49.189312 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:49.189289    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-02d29c-57f9f49485-4ks9w"]
Jul 09 11:51:50.950386 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:50.950352    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="828a337a-7887-4ea6-af95-ef2363365162" path="/var/lib/kubelet/pods/828a337a-7887-4ea6-af95-ef2363365162/volumes"
Jul 09 11:51:51.976494 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:51.976466    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:51:52.134096 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.134061    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-envoy\") pod \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") "
Jul 09 11:51:52.134278 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.134108    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-token\") pod \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") "
Jul 09 11:51:52.134278 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.134137    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-workload-socket\") pod \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") "
Jul 09 11:51:52.134278 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.134161    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istiod-ca-cert\") pod \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") "
Jul 09 11:51:52.134278 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.134209    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-podinfo\") pod \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") "
Jul 09 11:51:52.134278 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.134234    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-credential-socket\") pod \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") "
Jul 09 11:51:52.134557 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.134390    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvt7p\" (UniqueName: \"kubernetes.io/projected/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-kube-api-access-fvt7p\") pod \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") "
Jul 09 11:51:52.134557 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.134473    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-data\") pod \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") "
Jul 09 11:51:52.134557 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.134519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-workload-certs\") pod \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\" (UID: \"968ef205-fbe6-4947-ab53-51a3c4bb0ee6\") "
Jul 09 11:51:52.134704 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.134530    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "968ef205-fbe6-4947-ab53-51a3c4bb0ee6" (UID: "968ef205-fbe6-4947-ab53-51a3c4bb0ee6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:51:52.134704 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.134598    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "968ef205-fbe6-4947-ab53-51a3c4bb0ee6" (UID: "968ef205-fbe6-4947-ab53-51a3c4bb0ee6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:51:52.134868 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.134850    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:52.134950 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.134873    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:52.134950 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.134932    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "968ef205-fbe6-4947-ab53-51a3c4bb0ee6" (UID: "968ef205-fbe6-4947-ab53-51a3c4bb0ee6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:51:52.135120 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.135089    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "968ef205-fbe6-4947-ab53-51a3c4bb0ee6" (UID: "968ef205-fbe6-4947-ab53-51a3c4bb0ee6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:51:52.135575 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.135553    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-data" (OuterVolumeSpecName: "istio-data") pod "968ef205-fbe6-4947-ab53-51a3c4bb0ee6" (UID: "968ef205-fbe6-4947-ab53-51a3c4bb0ee6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:51:52.136416 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.136385    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "968ef205-fbe6-4947-ab53-51a3c4bb0ee6" (UID: "968ef205-fbe6-4947-ab53-51a3c4bb0ee6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:51:52.136529 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.136425    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-token" (OuterVolumeSpecName: "istio-token") pod "968ef205-fbe6-4947-ab53-51a3c4bb0ee6" (UID: "968ef205-fbe6-4947-ab53-51a3c4bb0ee6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:51:52.136920 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.136896    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "968ef205-fbe6-4947-ab53-51a3c4bb0ee6" (UID: "968ef205-fbe6-4947-ab53-51a3c4bb0ee6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 11:51:52.136970 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.136896    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-kube-api-access-fvt7p" (OuterVolumeSpecName: "kube-api-access-fvt7p") pod "968ef205-fbe6-4947-ab53-51a3c4bb0ee6" (UID: "968ef205-fbe6-4947-ab53-51a3c4bb0ee6"). InnerVolumeSpecName "kube-api-access-fvt7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:51:52.181018 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.180983    2563 generic.go:356] "Generic (PLEG): container finished" podID="968ef205-fbe6-4947-ab53-51a3c4bb0ee6" containerID="c3c6eb2ff88202d61979f0533513d00d65d4fe5a3712a1d597cf3aa87ffe3e23" exitCode=0
Jul 09 11:51:52.181171 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.181049    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"
Jul 09 11:51:52.181171 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.181071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w" event={"ID":"968ef205-fbe6-4947-ab53-51a3c4bb0ee6","Type":"ContainerDied","Data":"c3c6eb2ff88202d61979f0533513d00d65d4fe5a3712a1d597cf3aa87ffe3e23"}
Jul 09 11:51:52.181171 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.181135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w" event={"ID":"968ef205-fbe6-4947-ab53-51a3c4bb0ee6","Type":"ContainerDied","Data":"d6954d8e08580a564344d146388b0d4e220475fe9b713a2c8aafd2070603a58a"}
Jul 09 11:51:52.181273 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.181172    2563 scope.go:117] "RemoveContainer" containerID="c3c6eb2ff88202d61979f0533513d00d65d4fe5a3712a1d597cf3aa87ffe3e23"
Jul 09 11:51:52.190064 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.190045    2563 scope.go:117] "RemoveContainer" containerID="e87de1858bc28d4fdae7db93b0aae04ba28ac7edcc5fdfe6658c1ebc1853e24b"
Jul 09 11:51:52.197929 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.197907    2563 scope.go:117] "RemoveContainer" containerID="c3c6eb2ff88202d61979f0533513d00d65d4fe5a3712a1d597cf3aa87ffe3e23"
Jul 09 11:51:52.198208 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:51:52.198192    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c6eb2ff88202d61979f0533513d00d65d4fe5a3712a1d597cf3aa87ffe3e23\": container with ID starting with c3c6eb2ff88202d61979f0533513d00d65d4fe5a3712a1d597cf3aa87ffe3e23 not found: ID does not exist" containerID="c3c6eb2ff88202d61979f0533513d00d65d4fe5a3712a1d597cf3aa87ffe3e23"
Jul 09 11:51:52.198247 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.198216    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c6eb2ff88202d61979f0533513d00d65d4fe5a3712a1d597cf3aa87ffe3e23"} err="failed to get container status \"c3c6eb2ff88202d61979f0533513d00d65d4fe5a3712a1d597cf3aa87ffe3e23\": rpc error: code = NotFound desc = could not find container \"c3c6eb2ff88202d61979f0533513d00d65d4fe5a3712a1d597cf3aa87ffe3e23\": container with ID starting with c3c6eb2ff88202d61979f0533513d00d65d4fe5a3712a1d597cf3aa87ffe3e23 not found: ID does not exist"
Jul 09 11:51:52.198247 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.198232    2563 scope.go:117] "RemoveContainer" containerID="e87de1858bc28d4fdae7db93b0aae04ba28ac7edcc5fdfe6658c1ebc1853e24b"
Jul 09 11:51:52.198456 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:51:52.198439    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e87de1858bc28d4fdae7db93b0aae04ba28ac7edcc5fdfe6658c1ebc1853e24b\": container with ID starting with e87de1858bc28d4fdae7db93b0aae04ba28ac7edcc5fdfe6658c1ebc1853e24b not found: ID does not exist" containerID="e87de1858bc28d4fdae7db93b0aae04ba28ac7edcc5fdfe6658c1ebc1853e24b"
Jul 09 11:51:52.198492 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.198462    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e87de1858bc28d4fdae7db93b0aae04ba28ac7edcc5fdfe6658c1ebc1853e24b"} err="failed to get container status \"e87de1858bc28d4fdae7db93b0aae04ba28ac7edcc5fdfe6658c1ebc1853e24b\": rpc error: code = NotFound desc = could not find container \"e87de1858bc28d4fdae7db93b0aae04ba28ac7edcc5fdfe6658c1ebc1853e24b\": container with ID starting with e87de1858bc28d4fdae7db93b0aae04ba28ac7edcc5fdfe6658c1ebc1853e24b not found: ID does not exist"
Jul 09 11:51:52.205307 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.205287    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"]
Jul 09 11:51:52.209683 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.209657    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-a3dd9b-openshift-default-59db48bc75-9x47w"]
Jul 09 11:51:52.235605 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.235576    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:52.235605 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.235600    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:52.235605 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.235609    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:52.235778 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.235618    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:52.235778 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.235627    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvt7p\" (UniqueName: \"kubernetes.io/projected/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-kube-api-access-fvt7p\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:52.235778 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.235637    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:52.235778 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.235644    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/968ef205-fbe6-4947-ab53-51a3c4bb0ee6-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:52.950426 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:52.950397    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="968ef205-fbe6-4947-ab53-51a3c4bb0ee6" path="/var/lib/kubelet/pods/968ef205-fbe6-4947-ab53-51a3c4bb0ee6/volumes"
Jul 09 11:51:53.611198 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:53.611155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:53.711383 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:53.711352    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:53.882927 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:53.882838    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"]
Jul 09 11:51:53.883137 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:53.883110    2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq" podUID="d6cece14-7622-4e7d-a754-9b606afba55f" containerName="istio-proxy" containerID="cri-o://fbb53f68f00635763f694e27f02f5d67fb3574afc919545d8375b2e2d3a3fb0f" gracePeriod=30
Jul 09 11:51:53.894293 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:53.894268    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:51:56.977734 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:56.977703    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:51:56.984122 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:56.984098    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8x75w_75dd42d9-d05a-44c9-8d18-664ecaf7feeb/ovn-acl-logging/0.log"
Jul 09 11:51:59.123662 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.123639    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:51:59.213385 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.213303    2563 generic.go:356] "Generic (PLEG): container finished" podID="d6cece14-7622-4e7d-a754-9b606afba55f" containerID="fbb53f68f00635763f694e27f02f5d67fb3574afc919545d8375b2e2d3a3fb0f" exitCode=0
Jul 09 11:51:59.213385 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.213340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq" event={"ID":"d6cece14-7622-4e7d-a754-9b606afba55f","Type":"ContainerDied","Data":"fbb53f68f00635763f694e27f02f5d67fb3574afc919545d8375b2e2d3a3fb0f"}
Jul 09 11:51:59.213385 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.213356    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"
Jul 09 11:51:59.213385 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.213384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq" event={"ID":"d6cece14-7622-4e7d-a754-9b606afba55f","Type":"ContainerDied","Data":"2b2f8dd9649e85bc358475a4ae1edf8f2e9717e06736b83c8a8621d5bd19cc7f"}
Jul 09 11:51:59.213649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.213407    2563 scope.go:117] "RemoveContainer" containerID="fbb53f68f00635763f694e27f02f5d67fb3574afc919545d8375b2e2d3a3fb0f"
Jul 09 11:51:59.221343 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.221322    2563 scope.go:117] "RemoveContainer" containerID="c364eda291fd41f9fec780c141f676f3d443ecb9a5a8274377ab1ba56c2fef08"
Jul 09 11:51:59.228898 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.228877    2563 scope.go:117] "RemoveContainer" containerID="fbb53f68f00635763f694e27f02f5d67fb3574afc919545d8375b2e2d3a3fb0f"
Jul 09 11:51:59.229170 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:51:59.229151    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb53f68f00635763f694e27f02f5d67fb3574afc919545d8375b2e2d3a3fb0f\": container with ID starting with fbb53f68f00635763f694e27f02f5d67fb3574afc919545d8375b2e2d3a3fb0f not found: ID does not exist" containerID="fbb53f68f00635763f694e27f02f5d67fb3574afc919545d8375b2e2d3a3fb0f"
Jul 09 11:51:59.229214 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.229180    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb53f68f00635763f694e27f02f5d67fb3574afc919545d8375b2e2d3a3fb0f"} err="failed to get container status \"fbb53f68f00635763f694e27f02f5d67fb3574afc919545d8375b2e2d3a3fb0f\": rpc error: code = NotFound desc = could not find container \"fbb53f68f00635763f694e27f02f5d67fb3574afc919545d8375b2e2d3a3fb0f\": container with ID starting with fbb53f68f00635763f694e27f02f5d67fb3574afc919545d8375b2e2d3a3fb0f not found: ID does not exist"
Jul 09 11:51:59.229214 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.229198    2563 scope.go:117] "RemoveContainer" containerID="c364eda291fd41f9fec780c141f676f3d443ecb9a5a8274377ab1ba56c2fef08"
Jul 09 11:51:59.229450 ip-10-0-138-176 kubenswrapper[2563]: E0709 11:51:59.229436    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c364eda291fd41f9fec780c141f676f3d443ecb9a5a8274377ab1ba56c2fef08\": container with ID starting with c364eda291fd41f9fec780c141f676f3d443ecb9a5a8274377ab1ba56c2fef08 not found: ID does not exist" containerID="c364eda291fd41f9fec780c141f676f3d443ecb9a5a8274377ab1ba56c2fef08"
Jul 09 11:51:59.229499 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.229452    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c364eda291fd41f9fec780c141f676f3d443ecb9a5a8274377ab1ba56c2fef08"} err="failed to get container status \"c364eda291fd41f9fec780c141f676f3d443ecb9a5a8274377ab1ba56c2fef08\": rpc error: code = NotFound desc = could not find container \"c364eda291fd41f9fec780c141f676f3d443ecb9a5a8274377ab1ba56c2fef08\": container with ID starting with c364eda291fd41f9fec780c141f676f3d443ecb9a5a8274377ab1ba56c2fef08 not found: ID does not exist"
Jul 09 11:51:59.298703 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.298668    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-workload-certs\") pod \"d6cece14-7622-4e7d-a754-9b606afba55f\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") "
Jul 09 11:51:59.298879 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.298720    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6cece14-7622-4e7d-a754-9b606afba55f-istio-token\") pod \"d6cece14-7622-4e7d-a754-9b606afba55f\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") "
Jul 09 11:51:59.298879 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.298745    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6cece14-7622-4e7d-a754-9b606afba55f-istiod-ca-cert\") pod \"d6cece14-7622-4e7d-a754-9b606afba55f\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") "
Jul 09 11:51:59.298879 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.298766    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-workload-socket\") pod \"d6cece14-7622-4e7d-a754-9b606afba55f\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") "
Jul 09 11:51:59.298879 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.298825    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6cece14-7622-4e7d-a754-9b606afba55f-istio-podinfo\") pod \"d6cece14-7622-4e7d-a754-9b606afba55f\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") "
Jul 09 11:51:59.298879 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.298867    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-credential-socket\") pod \"d6cece14-7622-4e7d-a754-9b606afba55f\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") "
Jul 09 11:51:59.299096 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.298929    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-istio-envoy\") pod \"d6cece14-7622-4e7d-a754-9b606afba55f\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") "
Jul 09 11:51:59.299096 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.298958    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-istio-data\") pod \"d6cece14-7622-4e7d-a754-9b606afba55f\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") "
Jul 09 11:51:59.299096 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.299009    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58mr2\" (UniqueName: \"kubernetes.io/projected/d6cece14-7622-4e7d-a754-9b606afba55f-kube-api-access-58mr2\") pod \"d6cece14-7622-4e7d-a754-9b606afba55f\" (UID: \"d6cece14-7622-4e7d-a754-9b606afba55f\") "
Jul 09 11:51:59.299096 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.299019    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d6cece14-7622-4e7d-a754-9b606afba55f" (UID: "d6cece14-7622-4e7d-a754-9b606afba55f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:51:59.299296 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.299155    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d6cece14-7622-4e7d-a754-9b606afba55f" (UID: "d6cece14-7622-4e7d-a754-9b606afba55f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:51:59.299296 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.299251    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-workload-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:59.299296 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.299271    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-workload-certs\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:59.299296 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.299193    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6cece14-7622-4e7d-a754-9b606afba55f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d6cece14-7622-4e7d-a754-9b606afba55f" (UID: "d6cece14-7622-4e7d-a754-9b606afba55f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 11:51:59.299509 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.299401    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d6cece14-7622-4e7d-a754-9b606afba55f" (UID: "d6cece14-7622-4e7d-a754-9b606afba55f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:51:59.299801 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.299757    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-istio-data" (OuterVolumeSpecName: "istio-data") pod "d6cece14-7622-4e7d-a754-9b606afba55f" (UID: "d6cece14-7622-4e7d-a754-9b606afba55f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:51:59.301134 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.301108    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d6cece14-7622-4e7d-a754-9b606afba55f" (UID: "d6cece14-7622-4e7d-a754-9b606afba55f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 11:51:59.301134 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.301111    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6cece14-7622-4e7d-a754-9b606afba55f-istio-token" (OuterVolumeSpecName: "istio-token") pod "d6cece14-7622-4e7d-a754-9b606afba55f" (UID: "d6cece14-7622-4e7d-a754-9b606afba55f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:51:59.301269 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.301143    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6cece14-7622-4e7d-a754-9b606afba55f-kube-api-access-58mr2" (OuterVolumeSpecName: "kube-api-access-58mr2") pod "d6cece14-7622-4e7d-a754-9b606afba55f" (UID: "d6cece14-7622-4e7d-a754-9b606afba55f"). InnerVolumeSpecName "kube-api-access-58mr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 11:51:59.301269 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.301228    2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6cece14-7622-4e7d-a754-9b606afba55f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d6cece14-7622-4e7d-a754-9b606afba55f" (UID: "d6cece14-7622-4e7d-a754-9b606afba55f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 11:51:59.399974 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.399925    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-istio-envoy\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:59.399974 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.399971    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-istio-data\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:59.399974 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.399981    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58mr2\" (UniqueName: \"kubernetes.io/projected/d6cece14-7622-4e7d-a754-9b606afba55f-kube-api-access-58mr2\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:59.399974 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.399990    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6cece14-7622-4e7d-a754-9b606afba55f-istio-token\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:59.400253 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.400000    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6cece14-7622-4e7d-a754-9b606afba55f-istiod-ca-cert\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:59.400253 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.400007    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6cece14-7622-4e7d-a754-9b606afba55f-istio-podinfo\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:59.400253 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.400016    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6cece14-7622-4e7d-a754-9b606afba55f-credential-socket\") on node \"ip-10-0-138-176.ec2.internal\" DevicePath \"\""
Jul 09 11:51:59.540293 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.540258    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"]
Jul 09 11:51:59.544407 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.544377    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-02d29c-openshift-default-745ddbfb9f-xrcqq"]
Jul 09 11:51:59.921101 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:51:59.921027    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59bbb887b4-t7xfq_dbe83611-f892-4f10-9263-0c654cb10c3c/manager/0.log"
Jul 09 11:52:00.290930 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:00.290908    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65b4d55d4c-9vnpk_92da1819-00ee-4b3f-b105-25428d8fda49/copy-manifests/0.log"
Jul 09 11:52:00.298374 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:00.298351    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65b4d55d4c-9vnpk_92da1819-00ee-4b3f-b105-25428d8fda49/manager/0.log"
Jul 09 11:52:00.949652 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:00.949605    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6cece14-7622-4e7d-a754-9b606afba55f" path="/var/lib/kubelet/pods/d6cece14-7622-4e7d-a754-9b606afba55f/volumes"
Jul 09 11:52:03.307138 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:03.307106    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hm9wl_a7582b03-1757-439f-ae07-2112955e472e/discovery/0.log"
Jul 09 11:52:04.403445 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:04.403415    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-rjh6p_4ebc912d-4e28-44e9-a7a7-c399ee8419ee/main/0.log"
Jul 09 11:52:04.759970 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:04.759943    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dblkwh4_29a7d16f-5bda-42cf-a856-8c8cb67cd6ec/main/0.log"
Jul 09 11:52:09.751584 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:09.751550    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:52:09.848201 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:09.848167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log" limit=1024
Jul 09 11:52:12.768407 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:12.768379    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q7c8l_346f1026-54ad-4ae6-9134-e2d1a31e565f/global-pull-secret-syncer/0.log"
Jul 09 11:52:12.880913 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:12.880882    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2dkxw_0aee5e83-b717-46bf-b4d8-07a090cde2c3/konnectivity-agent/0.log"
Jul 09 11:52:13.007021 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:13.006982    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-176.ec2.internal_4aca4cc90e184685e6cdf76e3dd114f1/haproxy/0.log"
Jul 09 11:52:20.524009 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:20.523982    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-67b697bddd-26n9n_801b8517-ff16-41b5-a455-24fd8e17d0d3/metrics-server/0.log"
Jul 09 11:52:20.587128 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:20.587037    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8pslv_8176269c-37d1-47ed-9d2d-a7c49244ebe4/node-exporter/0.log"
Jul 09 11:52:20.610792 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:20.610750    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8pslv_8176269c-37d1-47ed-9d2d-a7c49244ebe4/kube-rbac-proxy/0.log"
Jul 09 11:52:20.636830 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:20.636807    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8pslv_8176269c-37d1-47ed-9d2d-a7c49244ebe4/init-textfile/0.log"
Jul 09 11:52:21.181761 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:21.181733    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6447f946f9-vgk4k_bd31e459-5084-4036-be6b-8690ad6879f0/telemeter-client/0.log"
Jul 09 11:52:21.226646 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:21.226620    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6447f946f9-vgk4k_bd31e459-5084-4036-be6b-8690ad6879f0/reload/0.log"
Jul 09 11:52:21.246849 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:21.246826    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6447f946f9-vgk4k_bd31e459-5084-4036-be6b-8690ad6879f0/kube-rbac-proxy/0.log"
Jul 09 11:52:23.118763 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.118730    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"]
Jul 09 11:52:23.119184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119046    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6cece14-7622-4e7d-a754-9b606afba55f" containerName="istio-proxy"
Jul 09 11:52:23.119184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119058    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6cece14-7622-4e7d-a754-9b606afba55f" containerName="istio-proxy"
Jul 09 11:52:23.119184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119073    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6cece14-7622-4e7d-a754-9b606afba55f" containerName="istio-proxy"
Jul 09 11:52:23.119184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119079    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6cece14-7622-4e7d-a754-9b606afba55f" containerName="istio-proxy"
Jul 09 11:52:23.119184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119087    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="968ef205-fbe6-4947-ab53-51a3c4bb0ee6" containerName="istio-proxy"
Jul 09 11:52:23.119184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119094    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="968ef205-fbe6-4947-ab53-51a3c4bb0ee6" containerName="istio-proxy"
Jul 09 11:52:23.119184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119099    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="828a337a-7887-4ea6-af95-ef2363365162" containerName="maas-api"
Jul 09 11:52:23.119184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119104    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="828a337a-7887-4ea6-af95-ef2363365162" containerName="maas-api"
Jul 09 11:52:23.119184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119114    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f9f94b7-0542-4a24-952b-c63ce6bb2eef" containerName="maas-api"
Jul 09 11:52:23.119184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119120    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f9f94b7-0542-4a24-952b-c63ce6bb2eef" containerName="maas-api"
Jul 09 11:52:23.119184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119126    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="968ef205-fbe6-4947-ab53-51a3c4bb0ee6" containerName="istio-proxy"
Jul 09 11:52:23.119184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119133    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="968ef205-fbe6-4947-ab53-51a3c4bb0ee6" containerName="istio-proxy"
Jul 09 11:52:23.119184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119180    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="968ef205-fbe6-4947-ab53-51a3c4bb0ee6" containerName="istio-proxy"
Jul 09 11:52:23.119184 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119188    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6cece14-7622-4e7d-a754-9b606afba55f" containerName="istio-proxy"
Jul 09 11:52:23.119630 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119196    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="828a337a-7887-4ea6-af95-ef2363365162" containerName="maas-api"
Jul 09 11:52:23.119630 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119201    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6cece14-7622-4e7d-a754-9b606afba55f" containerName="istio-proxy"
Jul 09 11:52:23.119630 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.119206    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f9f94b7-0542-4a24-952b-c63ce6bb2eef" containerName="maas-api"
Jul 09 11:52:23.122373 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.122348    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.124918 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.124895    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8tktg\"/\"openshift-service-ca.crt\""
Jul 09 11:52:23.125043 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.124939    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8tktg\"/\"default-dockercfg-rl825\""
Jul 09 11:52:23.125043 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.124963    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8tktg\"/\"kube-root-ca.crt\""
Jul 09 11:52:23.132116 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.132096    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"]
Jul 09 11:52:23.211263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.211230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9d5f78d-77fb-46c5-a838-ce74b96083c1-sys\") pod \"perf-node-gather-daemonset-ljch8\" (UID: \"d9d5f78d-77fb-46c5-a838-ce74b96083c1\") " pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.211263 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.211263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9d5f78d-77fb-46c5-a838-ce74b96083c1-podres\") pod \"perf-node-gather-daemonset-ljch8\" (UID: \"d9d5f78d-77fb-46c5-a838-ce74b96083c1\") " pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.211469 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.211291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqkwx\" (UniqueName: \"kubernetes.io/projected/d9d5f78d-77fb-46c5-a838-ce74b96083c1-kube-api-access-bqkwx\") pod \"perf-node-gather-daemonset-ljch8\" (UID: \"d9d5f78d-77fb-46c5-a838-ce74b96083c1\") " pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.211469 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.211316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9d5f78d-77fb-46c5-a838-ce74b96083c1-proc\") pod \"perf-node-gather-daemonset-ljch8\" (UID: \"d9d5f78d-77fb-46c5-a838-ce74b96083c1\") " pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.211469 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.211339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9d5f78d-77fb-46c5-a838-ce74b96083c1-lib-modules\") pod \"perf-node-gather-daemonset-ljch8\" (UID: \"d9d5f78d-77fb-46c5-a838-ce74b96083c1\") " pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.312479 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.312451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqkwx\" (UniqueName: \"kubernetes.io/projected/d9d5f78d-77fb-46c5-a838-ce74b96083c1-kube-api-access-bqkwx\") pod \"perf-node-gather-daemonset-ljch8\" (UID: \"d9d5f78d-77fb-46c5-a838-ce74b96083c1\") " pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.312660 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.312485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9d5f78d-77fb-46c5-a838-ce74b96083c1-proc\") pod \"perf-node-gather-daemonset-ljch8\" (UID: \"d9d5f78d-77fb-46c5-a838-ce74b96083c1\") " pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.312660 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.312513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9d5f78d-77fb-46c5-a838-ce74b96083c1-lib-modules\") pod \"perf-node-gather-daemonset-ljch8\" (UID: \"d9d5f78d-77fb-46c5-a838-ce74b96083c1\") " pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.312660 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.312596    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9d5f78d-77fb-46c5-a838-ce74b96083c1-proc\") pod \"perf-node-gather-daemonset-ljch8\" (UID: \"d9d5f78d-77fb-46c5-a838-ce74b96083c1\") " pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.312660 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.312621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9d5f78d-77fb-46c5-a838-ce74b96083c1-sys\") pod \"perf-node-gather-daemonset-ljch8\" (UID: \"d9d5f78d-77fb-46c5-a838-ce74b96083c1\") " pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.312660 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.312644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9d5f78d-77fb-46c5-a838-ce74b96083c1-podres\") pod \"perf-node-gather-daemonset-ljch8\" (UID: \"d9d5f78d-77fb-46c5-a838-ce74b96083c1\") " pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.312887 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.312693    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9d5f78d-77fb-46c5-a838-ce74b96083c1-sys\") pod \"perf-node-gather-daemonset-ljch8\" (UID: \"d9d5f78d-77fb-46c5-a838-ce74b96083c1\") " pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.312887 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.312737    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9d5f78d-77fb-46c5-a838-ce74b96083c1-podres\") pod \"perf-node-gather-daemonset-ljch8\" (UID: \"d9d5f78d-77fb-46c5-a838-ce74b96083c1\") " pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.312887 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.312736    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9d5f78d-77fb-46c5-a838-ce74b96083c1-lib-modules\") pod \"perf-node-gather-daemonset-ljch8\" (UID: \"d9d5f78d-77fb-46c5-a838-ce74b96083c1\") " pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.321535 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.321510    2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqkwx\" (UniqueName: \"kubernetes.io/projected/d9d5f78d-77fb-46c5-a838-ce74b96083c1-kube-api-access-bqkwx\") pod \"perf-node-gather-daemonset-ljch8\" (UID: \"d9d5f78d-77fb-46c5-a838-ce74b96083c1\") " pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.432957 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.432867    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:23.558428 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:23.558401    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"]
Jul 09 11:52:23.561197 ip-10-0-138-176 kubenswrapper[2563]: W0709 11:52:23.561166    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd9d5f78d_77fb_46c5_a838_ce74b96083c1.slice/crio-0321824cb60fb8e58e8905f85a4ff46ad3de1c9dc7326ffa870c3f869fdf4599 WatchSource:0}: Error finding container 0321824cb60fb8e58e8905f85a4ff46ad3de1c9dc7326ffa870c3f869fdf4599: Status 404 returned error can't find the container with id 0321824cb60fb8e58e8905f85a4ff46ad3de1c9dc7326ffa870c3f869fdf4599
Jul 09 11:52:24.320396 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:24.320358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8" event={"ID":"d9d5f78d-77fb-46c5-a838-ce74b96083c1","Type":"ContainerStarted","Data":"6765dd50537471b6dfbe2e7b339cbf8a77436ba18006eda2c8662f19b7c51a99"}
Jul 09 11:52:24.320828 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:24.320414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8" event={"ID":"d9d5f78d-77fb-46c5-a838-ce74b96083c1","Type":"ContainerStarted","Data":"0321824cb60fb8e58e8905f85a4ff46ad3de1c9dc7326ffa870c3f869fdf4599"}
Jul 09 11:52:24.320828 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:24.320451    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:24.337480 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:24.337426    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8" podStartSLOduration=1.3374072830000001 podStartE2EDuration="1.337407283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 11:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 11:52:24.336818057 +0000 UTC m=+2427.928549219" watchObservedRunningTime="2026-07-09 11:52:24.337407283 +0000 UTC m=+2427.929138450"
Jul 09 11:52:25.710481 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:25.710452    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fmjjv_49b90316-83e9-4bc1-a9b3-e118c434d9fd/dns/0.log"
Jul 09 11:52:25.733489 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:25.733461    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fmjjv_49b90316-83e9-4bc1-a9b3-e118c434d9fd/kube-rbac-proxy/0.log"
Jul 09 11:52:25.895384 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:25.895346    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k7d7z_72b5c8e9-cd8b-4dde-9850-824fbf0818e7/dns-node-resolver/0.log"
Jul 09 11:52:26.522976 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:26.522951    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tx572_00d1d258-b95a-4f9e-88ac-ac35af980a14/node-ca/0.log"
Jul 09 11:52:27.583833 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:27.583802    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hm9wl_a7582b03-1757-439f-ae07-2112955e472e/discovery/0.log"
Jul 09 11:52:28.538088 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:28.538049    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-745gr_53a84cad-5a9e-4284-95f4-7ed61d11ac64/serve-healthcheck-canary/0.log"
Jul 09 11:52:29.127944 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:29.127918    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gmlwh_5569f56e-a640-4762-b216-9dbd9e18f3f5/kube-rbac-proxy/0.log"
Jul 09 11:52:29.151125 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:29.151102    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gmlwh_5569f56e-a640-4762-b216-9dbd9e18f3f5/exporter/0.log"
Jul 09 11:52:29.172865 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:29.172839    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gmlwh_5569f56e-a640-4762-b216-9dbd9e18f3f5/extractor/0.log"
Jul 09 11:52:30.334042 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:30.334017    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8tktg/perf-node-gather-daemonset-ljch8"
Jul 09 11:52:31.938649 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:31.938618    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59bbb887b4-t7xfq_dbe83611-f892-4f10-9263-0c654cb10c3c/manager/0.log"
Jul 09 11:52:32.072222 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:32.072191    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65b4d55d4c-9vnpk_92da1819-00ee-4b3f-b105-25428d8fda49/manager/0.log"
Jul 09 11:52:32.097713 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:32.097684    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65b4d55d4c-9vnpk_92da1819-00ee-4b3f-b105-25428d8fda49/copy-manifests/0.log"
Jul 09 11:52:33.752585 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:33.752559    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-cbdc75774-xf7gg_e71125e1-b9e6-42b6-855c-f256a120169c/manager/0.log"
Jul 09 11:52:34.140687 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:34.140659    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hm9wl_a7582b03-1757-439f-ae07-2112955e472e/discovery/0.log"
Jul 09 11:52:35.640686 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:35.640655    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hm9wl_a7582b03-1757-439f-ae07-2112955e472e/discovery/0.log"
Jul 09 11:52:40.950323 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:40.950292    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8rzgk_f1907393-916d-43d8-8ec1-0cb8029ad259/kube-multus-additional-cni-plugins/0.log"
Jul 09 11:52:40.973457 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:40.973434    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8rzgk_f1907393-916d-43d8-8ec1-0cb8029ad259/egress-router-binary-copy/0.log"
Jul 09 11:52:40.995335 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:40.995312    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8rzgk_f1907393-916d-43d8-8ec1-0cb8029ad259/cni-plugins/0.log"
Jul 09 11:52:41.019461 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:41.019430    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8rzgk_f1907393-916d-43d8-8ec1-0cb8029ad259/bond-cni-plugin/0.log"
Jul 09 11:52:41.041895 ip-10-0-138-176 kubenswrapper[2563]: I0709 11:52:41.041864    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8rzgk_f1907393-916d-43d8-8ec1-0cb8029ad259/routeoverride-cni/0.log"