Jul 14 00:57:43.051499 ip-10-0-129-117 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 14 00:57:43.051509 ip-10-0-129-117 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 14 00:57:43.051517 ip-10-0-129-117 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 14 00:57:43.051788 ip-10-0-129-117 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 14 00:57:53.111442 ip-10-0-129-117 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 14 00:57:53.111455 ip-10-0-129-117 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 816a20e4d2d14b4c8df871586af1661f --
Jul 14 00:59:30.289290 ip-10-0-129-117 systemd[1]: Starting Kubernetes Kubelet...
Jul 14 00:59:30.760440 ip-10-0-129-117 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 00:59:30.760440 ip-10-0-129-117 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 14 00:59:30.760440 ip-10-0-129-117 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 00:59:30.760440 ip-10-0-129-117 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 14 00:59:30.760440 ip-10-0-129-117 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 00:59:30.761147 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.760527    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 14 00:59:30.764986 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.764949    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:59:30.764986 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.764985    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:59:30.764986 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.764989    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.764994    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.764997    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765000    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765003    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765006    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765008    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765011    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765013    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765016    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765018    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765021    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765023    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765026    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765028    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765031    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765033    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765036    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765043    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765045    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:59:30.765088 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765048    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765103    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765108    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765111    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765114    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765117    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765120    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765123    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765125    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765130    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765133    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765136    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765139    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765142    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765146    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765148    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765151    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765153    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765157    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765160    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:59:30.765560 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765162    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765166    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765169    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765173    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765176    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765179    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765181    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765183    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765186    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765188    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765191    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765193    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765196    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765198    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765202    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765205    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765208    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765210    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765212    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:59:30.766070 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765215    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765217    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765220    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765222    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765226    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765229    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765231    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765234    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765237    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765240    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765242    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765246    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765250    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765253    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765256    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765259    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765262    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765264    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765267    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:59:30.766532 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765270    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765272    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765275    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765279    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765281    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765284    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765286    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765289    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765293    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765295    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765298    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765300    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765303    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765306    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765308    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765310    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765313    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765315    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:59:30.767012 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.765317    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767402    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767410    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767413    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767416    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767419    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767423    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767426    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767429    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767431    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767434    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767438    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767440    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767443    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767446    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767448    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767451    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767453    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767456    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767460    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:59:30.767487 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767463    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767466    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767469    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767471    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767474    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767477    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767480    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767482    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767484    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767487    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767491    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767495    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767498    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767501    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767504    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767506    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767509    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767512    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767514    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767519    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:59:30.768023 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767522    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767524    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767527    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767529    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767532    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767534    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767536    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767539    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767541    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767544    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767547    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767549    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767551    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767554    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767558    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767561    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767563    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767566    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767568    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767571    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:59:30.768514 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767573    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767576    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767578    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767581    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767584    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767586    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767589    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767591    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767594    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767596    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767599    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767601    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767606    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767610    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767613    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767615    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767619    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767622    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767626    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:59:30.769015 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767629    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767632    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767634    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767637    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767639    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767642    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767644    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767647    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767650    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767652    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767655    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767658    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767660    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767663    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767666    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767669    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767672    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767675    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767677    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767680    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:59:30.769480 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.767682    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768123    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768130    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768134    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768137    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768143    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768146    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768148    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768151    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768155    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768159    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768162    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768165    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768167    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768170    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768172    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768175    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768177    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768180    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:59:30.770029 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768184    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768187    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768190    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768193    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768195    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768198    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768200    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768203    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768206    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768208    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768210    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768213    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768215    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768218    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768220    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768223    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768225    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768227    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768230    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768233    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:59:30.770489 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768235    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768237    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768240    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768243    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768246    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768248    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768250    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768252    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768255    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768257    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768260    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768262    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768264    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768267    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768270    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768273    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768275    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768278    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768280    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:59:30.770988 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768282    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768285    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768287    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768290    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768293    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768296    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768299    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768301    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768303    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768306    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768309    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768311    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768314    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768318    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768320    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768323    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768325    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768327    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768330    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:59:30.771450 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768332    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768335    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768337    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768339    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768342    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768344    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768346    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768349    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768351    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768354    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768356    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768358    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768361    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768363    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768366    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768368    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768371    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768373    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768376    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768378    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:59:30.771922 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768380    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768383    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.768385    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768460    2567 flags.go:64] FLAG: --address="0.0.0.0"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768467    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768474    2567 flags.go:64] FLAG: --anonymous-auth="true"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768479    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768495    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768499    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768503    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768508    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768511    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768514    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768517    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768521    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768524    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768527    2567 flags.go:64] FLAG: --cgroup-root=""
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768530    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768533    2567 flags.go:64] FLAG: --client-ca-file=""
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768536    2567 flags.go:64] FLAG: --cloud-provider="external"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768539    2567 flags.go:64] FLAG: --cluster-dns="[]"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768543    2567 flags.go:64] FLAG: --cluster-domain=""
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768546    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768549    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 14 00:59:30.772457 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768552    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768556    2567 flags.go:64] FLAG: --container-log-max-files="5"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768559    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768562    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768565    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768568    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768572    2567 flags.go:64] FLAG: --contention-profiling="false"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768574    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768577    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768580    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768583    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768588    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768591    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768594    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768597    2567 flags.go:64] FLAG: --enable-load-reader="false"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768601    2567 flags.go:64] FLAG: --enable-server="true"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768604    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768609    2567 flags.go:64] FLAG: --event-burst="100"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768612    2567 flags.go:64] FLAG: --event-qps="50"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768615    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768618    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768621    2567 flags.go:64] FLAG: --eviction-hard=""
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768625    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768628    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768631    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 14 00:59:30.773047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768634    2567 flags.go:64] FLAG: --eviction-soft=""
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768637    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768639    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768642    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768645    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768650    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768653    2567 flags.go:64] FLAG: --fail-swap-on="true"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768656    2567 flags.go:64] FLAG: --feature-gates=""
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768660    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768663    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768666    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768669    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768672    2567 flags.go:64] FLAG: --healthz-port="10248"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768675    2567 flags.go:64] FLAG: --help="false"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768677    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-129-117.ec2.internal"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768681    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768683    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768686    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768690    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768693    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768696    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768698    2567 flags.go:64] FLAG: --image-service-endpoint=""
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768701    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 14 00:59:30.773638 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768704    2567 flags.go:64] FLAG: --kube-api-burst="100"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768707    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768711    2567 flags.go:64] FLAG: --kube-api-qps="50"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768713    2567 flags.go:64] FLAG: --kube-reserved=""
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768716    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768719    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768721    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768724    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768726    2567 flags.go:64] FLAG: --lock-file=""
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768729    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768732    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768735    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768740    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768743    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768747    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768750    2567 flags.go:64] FLAG: --logging-format="text"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768752    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768756    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768759    2567 flags.go:64] FLAG: --manifest-url=""
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768762    2567 flags.go:64] FLAG: --manifest-url-header=""
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768766    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768769    2567 flags.go:64] FLAG: --max-open-files="1000000"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768773    2567 flags.go:64] FLAG: --max-pods="110"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768776    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768779    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 14 00:59:30.774237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768782    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768784    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768789    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768792    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768794    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768802    2567 flags.go:64] FLAG: --node-status-max-images="50"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768805    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768808    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768811    2567 flags.go:64] FLAG: --pod-cidr=""
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768815    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768821    2567 flags.go:64] FLAG: --pod-manifest-path=""
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768823    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768827    2567 flags.go:64] FLAG: --pods-per-core="0"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768829    2567 flags.go:64] FLAG: --port="10250"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768832    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768835    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d06930c6786f87aa"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768838    2567 flags.go:64] FLAG: --qos-reserved=""
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768841    2567 flags.go:64] FLAG: --read-only-port="10255"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768844    2567 flags.go:64] FLAG: --register-node="true"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768847    2567 flags.go:64] FLAG: --register-with-taints=""
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768850    2567 flags.go:64] FLAG: --registry-burst="10"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768854    2567 flags.go:64] FLAG: --registry-qps="5"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768857    2567 flags.go:64] FLAG: --reserved-cpus=""
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768860    2567 flags.go:64] FLAG: --reserved-memory=""
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768864    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 14 00:59:30.774840 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768867    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768870    2567 flags.go:64] FLAG: --rotate-certificates="false"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768873    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768876    2567 flags.go:64] FLAG: --runonce="false"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768878    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768881    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768884    2567 flags.go:64] FLAG: --seccomp-default="false"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768887    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768890    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768894    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768897    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768900    2567 flags.go:64] FLAG: --storage-driver-password="root"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768903    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768906    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768909    2567 flags.go:64] FLAG: --storage-driver-user="root"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768911    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768914    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768918    2567 flags.go:64] FLAG: --system-cgroups=""
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768921    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768926    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768929    2567 flags.go:64] FLAG: --tls-cert-file=""
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768931    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768935    2567 flags.go:64] FLAG: --tls-min-version=""
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768938    2567 flags.go:64] FLAG: --tls-private-key-file=""
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768941    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 14 00:59:30.775530 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768943    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768946    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768949    2567 flags.go:64] FLAG: --v="2"
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768968    2567 flags.go:64] FLAG: --version="false"
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768974    2567 flags.go:64] FLAG: --vmodule=""
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768978    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.768981    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769102    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769106    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769109    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769111    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769114    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769117    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769120    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769122    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769125    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769129    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769131    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769134    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769136    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769139    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769141    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:59:30.776205 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769144    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769147    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769149    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769152    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769155    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769157    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769160    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769162    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769165    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769167    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769169    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769172    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769174    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769177    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769180    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769183    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769185    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769187    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769190    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769192    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:59:30.776742 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769194    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769197    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769200    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769202    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769205    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769207    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769211    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769214    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769216    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769219    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769221    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769224    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769226    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769229    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769231    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769233    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769236    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769239    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769241    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769244    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:59:30.777270 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769246    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769248    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769252    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769256    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769259    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769262    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769265    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769268    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769270    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769273    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769275    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769278    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769280    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769284    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769287    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769290    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769293    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769296    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769300    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:59:30.777765 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769302    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769305    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769307    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769309    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769312    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769315    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769317    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769320    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769322    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769324    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769327    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769330    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769332    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769335    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769337    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769339    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769342    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769344    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769346    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769350    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:59:30.778271 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769352    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:59:30.778750 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769355    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:59:30.778750 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769358    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:59:30.778750 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769362    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:59:30.778750 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.769364    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:59:30.778750 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.769830    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 00:59:30.778750 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.776801    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 14 00:59:30.778750 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.776824    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 14 00:59:30.778750 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.777701    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 14 00:59:30.778750 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.777710    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 14 00:59:30.778750 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777774    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:59:30.778750 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777781    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:59:30.778750 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777784    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:59:30.778750 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777788    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:59:30.778750 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777791    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777794    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777796    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777799    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777802    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777805    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777809    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777811    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777814    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777817    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777820    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777823    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777825    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777828    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777831    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777833    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777836    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777838    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777841    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777843    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:59:30.779121 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777846    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777848    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777851    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777853    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777856    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777860    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777863    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777865    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777868    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777870    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777873    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777875    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777879    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777883    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777885    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777888    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777890    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777893    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777895    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:59:30.779600 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777898    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777900    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777903    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777906    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777908    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777911    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777914    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777916    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777919    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777922    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777924    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777927    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777929    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777932    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777934    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777937    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777939    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777942    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777945    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777949    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:59:30.780089 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777971    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777974    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777978    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777980    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777983    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777985    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777988    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777991    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777993    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777995    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.777998    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778000    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778003    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778006    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778010    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778013    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778016    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778019    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778022    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:59:30.780575 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778024    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778027    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778029    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778032    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778034    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778036    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778039    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778041    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778044    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778046    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778049    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778053    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778057    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778061    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778064    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778067    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:59:30.781184 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778069    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:59:30.781580 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.778075    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 00:59:30.781580 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778193    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:59:30.781580 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778199    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:59:30.781580 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778202    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:59:30.781580 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778205    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:59:30.781580 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778208    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:59:30.781580 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778210    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:59:30.781580 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778213    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:59:30.781580 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778216    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:59:30.781580 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778219    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:59:30.781580 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778222    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:59:30.781580 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778225    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:59:30.781580 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778227    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:59:30.781580 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778230    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:59:30.781580 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778232    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778235    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778237    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778240    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778243    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778245    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778247    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778250    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778252    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778255    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778257    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778260    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778262    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778265    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778269    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778274    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778277    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778280    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778283    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:59:30.781941 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778286    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778289    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778291    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778294    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778296    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778299    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778301    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778303    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778306    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778308    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778311    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778313    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778316    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778318    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778321    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778323    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778326    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778328    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778330    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778333    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:59:30.782439 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778335    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778337    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778340    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778342    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778345    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778348    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778350    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778353    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778355    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778359    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778361    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778364    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778366    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778369    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778371    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778374    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778376    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778379    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778381    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778383    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:59:30.782923 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778387    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778390    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778392    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778395    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778398    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778400    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778403    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778405    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778407    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778410    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778412    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778414    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778417    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778419    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778422    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778424    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778427    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778430    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778432    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:59:30.783424 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778436    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:59:30.783891 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778440    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:59:30.783891 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778443    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:59:30.783891 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778446    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:59:30.783891 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778448    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:59:30.783891 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778451    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:59:30.783891 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778453    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:59:30.783891 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:30.778455    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:59:30.783891 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.778461    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 00:59:30.783891 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.778601    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 14 00:59:30.783891 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.780724    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 14 00:59:30.783891 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.782160    2567 server.go:1019] "Starting client certificate rotation"
Jul 14 00:59:30.783891 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.782273    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 00:59:30.783891 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.782345    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 00:59:30.784910 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.784892    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 00:59:30.789971 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.789929    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 00:59:30.800014 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.799988    2567 log.go:25] "Validated CRI v1 runtime API"
Jul 14 00:59:30.801973 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.801940    2567 log.go:25] "Validated CRI v1 image API"
Jul 14 00:59:30.802886 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.802871    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 14 00:59:30.810069 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.810042    2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8b55f8b0-4389-4b6a-a08f-d9b343c9bd1a:/dev/nvme0n1p4 d4d1c1db-03cd-4812-9610-1a13bb522625:/dev/nvme0n1p3]
Jul 14 00:59:30.810147 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.810067    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 14 00:59:30.812688 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.812662    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 00:59:30.815360 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.815248    2567 manager.go:216] Machine: {Timestamp:2026-07-14 00:59:30.813892369 +0000 UTC m=+0.378301415 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099454 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27ac5e71e3e6f385d7fcf5d1097487 SystemUUID:ec27ac5e-71e3-e6f3-85d7-fcf5d1097487 BootID:816a20e4-d2d1-4b4c-8df8-71586af1661f Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:30:34:5c:a4:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:30:34:5c:a4:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:fb:d2:88:75:d3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 14 00:59:30.815360 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.815351    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 14 00:59:30.815495 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.815441    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 14 00:59:30.818847 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.818810    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 14 00:59:30.819014 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.818847    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-117.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 14 00:59:30.820140 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.820127    2567 topology_manager.go:138] "Creating topology manager with none policy"
Jul 14 00:59:30.820178 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.820142    2567 container_manager_linux.go:309] "Creating device plugin manager"
Jul 14 00:59:30.820178 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.820150    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 00:59:30.820178 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.820161    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 00:59:30.821080 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.821069    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 14 00:59:30.824005 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.823989    2567 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 00:59:30.824140 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.824130    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 14 00:59:30.824169 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.824163    2567 kubelet.go:486] "Attempting to sync node with API server"
Jul 14 00:59:30.824227 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.824188    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 14 00:59:30.824227 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.824203    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 14 00:59:30.824227 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.824213    2567 kubelet.go:398] "Adding apiserver pod source"
Jul 14 00:59:30.824227 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.824224    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 14 00:59:30.824625 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.824600    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 00:59:30.824625 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.824626    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 00:59:30.826977 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.826946    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 14 00:59:30.828522 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.828509    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 14 00:59:30.828566 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.828540    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 14 00:59:30.829181 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.829168    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 14 00:59:30.829220 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.829187    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 14 00:59:30.829220 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.829198    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 14 00:59:30.829220 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.829206    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 14 00:59:30.829220 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.829214    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 14 00:59:30.829220 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.829220    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 14 00:59:30.829353 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.829269    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 14 00:59:30.829353 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.829274    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 14 00:59:30.829353 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.829280    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 14 00:59:30.829353 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.829286    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 14 00:59:30.829353 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.829299    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 14 00:59:30.829353 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.829307    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 14 00:59:30.831157 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.831144    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 14 00:59:30.831205 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.831159    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 14 00:59:30.834651 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.834632    2567 server.go:1268] "Started kubelet"
Jul 14 00:59:30.834764 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.834703    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 14 00:59:30.834816 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.834732    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 14 00:59:30.834868 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.834818    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 14 00:59:30.835607 ip-10-0-129-117 systemd[1]: Started Kubernetes Kubelet.
Jul 14 00:59:30.837556 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:30.837528    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 14 00:59:30.837753 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.837710    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-117.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 14 00:59:30.838451 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:30.837824    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 14 00:59:30.839121 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.839097    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 14 00:59:30.840114 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.840097    2567 server.go:310] "Adding debug handlers to kubelet server"
Jul 14 00:59:30.841506 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:30.841485    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 14 00:59:30.843761 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.843742    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 14 00:59:30.843845 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.843744    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 14 00:59:30.844240 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:30.843835    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
Jul 14 00:59:30.844422 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.844408    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 14 00:59:30.844537 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.844518    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 14 00:59:30.844695 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.844410    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 14 00:59:30.844695 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.844626    2567 reconstruct.go:97] "Volume reconstruction finished"
Jul 14 00:59:30.844695 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.844635    2567 reconciler.go:29] "Reconciler: start to sync state"
Jul 14 00:59:30.844695 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.844638    2567 factory.go:55] Registering systemd factory
Jul 14 00:59:30.844695 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.844655    2567 factory.go:223] Registration of the systemd container factory successfully
Jul 14 00:59:30.844910 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:30.843448    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-117.ec2.internal.18c20227722bdfbb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-117.ec2.internal,UID:ip-10-0-129-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-117.ec2.internal,},FirstTimestamp:2026-07-14 00:59:30.834595771 +0000 UTC m=+0.399004817,LastTimestamp:2026-07-14 00:59:30.834595771 +0000 UTC m=+0.399004817,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-117.ec2.internal,}"
Jul 14 00:59:30.844910 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.844864    2567 factory.go:153] Registering CRI-O factory
Jul 14 00:59:30.844910 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.844875    2567 factory.go:223] Registration of the crio container factory successfully
Jul 14 00:59:30.845062 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.844919    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 14 00:59:30.845062 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.844942    2567 factory.go:103] Registering Raw factory
Jul 14 00:59:30.845062 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.844973    2567 manager.go:1196] Started watching for new ooms in manager
Jul 14 00:59:30.845438 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.845425    2567 manager.go:349] Starting recovery of all containers
Jul 14 00:59:30.853038 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:30.850318    2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 14 00:59:30.853038 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:30.850424    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 14 00:59:30.856238 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.856212    2567 manager.go:354] Recovery completed
Jul 14 00:59:30.861248 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.861232    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:59:30.862168 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.862150    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dgmht"
Jul 14 00:59:30.863187 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.863166    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:59:30.863289 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.863200    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:59:30.863289 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.863210    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:59:30.863856 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.863841    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 14 00:59:30.863856 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.863853    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 14 00:59:30.863936 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.863869    2567 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 00:59:30.863936 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.863876    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:59:30.864669 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:30.864589    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-117.ec2.internal.18c2022773e01ca0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-117.ec2.internal,UID:ip-10-0-129-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-117.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-117.ec2.internal,},FirstTimestamp:2026-07-14 00:59:30.863185056 +0000 UTC m=+0.427594105,LastTimestamp:2026-07-14 00:59:30.863185056 +0000 UTC m=+0.427594105,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-117.ec2.internal,}"
Jul 14 00:59:30.864759 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.864726    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:59:30.864759 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.864746    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:59:30.864759 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.864755    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:59:30.865793 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.865774    2567 policy_none.go:49] "None policy: Start"
Jul 14 00:59:30.865793 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.865792    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 14 00:59:30.865946 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.865807    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 14 00:59:30.867388 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.867374    2567 policy_none.go:47] "Start"
Jul 14 00:59:30.869589 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.869555    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dgmht"
Jul 14 00:59:30.875431 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:30.875361    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-117.ec2.internal.18c2022773e06934  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-117.ec2.internal,UID:ip-10-0-129-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-117.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-117.ec2.internal,},FirstTimestamp:2026-07-14 00:59:30.86320466 +0000 UTC m=+0.427613709,LastTimestamp:2026-07-14 00:59:30.86320466 +0000 UTC m=+0.427613709,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-117.ec2.internal,}"
Jul 14 00:59:30.924438 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.911737    2567 manager.go:337] "Starting Device Plugin manager"
Jul 14 00:59:30.924438 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:30.911967    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 14 00:59:30.924438 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.911984    2567 server.go:87] "Starting device plugin registration server"
Jul 14 00:59:30.924438 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.912253    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 14 00:59:30.924438 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.912263    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 14 00:59:30.924438 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.912348    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 14 00:59:30.924438 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.912461    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 14 00:59:30.924438 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.912471    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 14 00:59:30.924438 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:30.913024    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 14 00:59:30.924438 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:30.913080    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-117.ec2.internal\" not found"
Jul 14 00:59:30.966833 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.966793    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 14 00:59:30.968226 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.968204    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 14 00:59:30.968302 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.968254    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 14 00:59:30.968302 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.968285    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 14 00:59:30.968391 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:30.968332    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 14 00:59:30.970858 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:30.970835    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 00:59:31.012918 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.012805    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:59:31.013892 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.013875    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:59:31.014037 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.013912    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:59:31.014037 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.013926    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:59:31.014037 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.013974    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.023706 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.023677    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.023830 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:31.023711    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-117.ec2.internal\": node \"ip-10-0-129-117.ec2.internal\" not found"
Jul 14 00:59:31.048292 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:31.048257    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
Jul 14 00:59:31.068671 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.068639    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"]
Jul 14 00:59:31.068788 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.068713    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:59:31.069731 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.069711    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:59:31.069834 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.069748    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:59:31.069834 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.069760    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:59:31.071373 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.071355    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:59:31.071472 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.071455    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.071534 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.071484    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:59:31.072195 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.072179    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:59:31.072353 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.072210    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:59:31.072353 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.072223    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:59:31.072353 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.072180    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:59:31.072353 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.072283    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:59:31.072353 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.072296    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:59:31.073512 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.073495    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.073598 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.073527    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:59:31.074289 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.074273    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:59:31.074375 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.074305    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:59:31.074375 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.074319    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:59:31.097938 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:31.097910    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-117.ec2.internal\" not found" node="ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.101440 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:31.101424    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-117.ec2.internal\" not found" node="ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.146481 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.146447    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9105b408cddaedf77cc3bc80f328aa9e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-117.ec2.internal\" (UID: \"9105b408cddaedf77cc3bc80f328aa9e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.146481 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.146482    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00280b56ea67b292319742fa9e3451f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"00280b56ea67b292319742fa9e3451f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.146617 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.146501    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00280b56ea67b292319742fa9e3451f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"00280b56ea67b292319742fa9e3451f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.148540 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:31.148524    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
Jul 14 00:59:31.246754 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.246710    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9105b408cddaedf77cc3bc80f328aa9e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-117.ec2.internal\" (UID: \"9105b408cddaedf77cc3bc80f328aa9e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.246754 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.246750    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00280b56ea67b292319742fa9e3451f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"00280b56ea67b292319742fa9e3451f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.246947 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.246767    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00280b56ea67b292319742fa9e3451f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"00280b56ea67b292319742fa9e3451f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.246947 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.246797    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9105b408cddaedf77cc3bc80f328aa9e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-117.ec2.internal\" (UID: \"9105b408cddaedf77cc3bc80f328aa9e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.246947 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.246820    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00280b56ea67b292319742fa9e3451f2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"00280b56ea67b292319742fa9e3451f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.246947 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.246808    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00280b56ea67b292319742fa9e3451f2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"00280b56ea67b292319742fa9e3451f2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.248835 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:31.248816    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
Jul 14 00:59:31.349687 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:31.349625    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
Jul 14 00:59:31.400858 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.400829    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.405739 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.404974    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
Jul 14 00:59:31.450060 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:31.450019    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
Jul 14 00:59:31.550676 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:31.550639    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
Jul 14 00:59:31.651169 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:31.651134    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
Jul 14 00:59:31.751777 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:31.751739    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
Jul 14 00:59:31.781975 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.781913    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 14 00:59:31.782470 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.782161    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 00:59:31.852039 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:31.852005    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
Jul 14 00:59:31.875368 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.875320    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 00:54:31 +0000 UTC" deadline="2028-02-19 02:57:31.819350453 +0000 UTC"
Jul 14 00:59:31.875368 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.875363    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14041h57m59.943991704s"
Jul 14 00:59:31.953083 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:31.953005    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
Jul 14 00:59:31.958196 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.958172    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:59:31.959523 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.959501    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:59:31.959630 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.959543    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:59:31.959630 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.959560    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:59:31.969800 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:31.969755    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00280b56ea67b292319742fa9e3451f2.slice/crio-17c32e6723b809c7d8d97f7c70286c3c664701a2865dfe2844a1400ea9bcb120 WatchSource:0}: Error finding container 17c32e6723b809c7d8d97f7c70286c3c664701a2865dfe2844a1400ea9bcb120: Status 404 returned error can't find the container with id 17c32e6723b809c7d8d97f7c70286c3c664701a2865dfe2844a1400ea9bcb120
Jul 14 00:59:31.975343 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:31.975321    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 00:59:32.053588 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:32.053540    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
Jul 14 00:59:32.056284 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.056259    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 00:59:32.069509 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:32.069482    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9105b408cddaedf77cc3bc80f328aa9e.slice/crio-c84a545afdf1d3e3e0f8ae99f2e20bc193fc01572486611bbb15f3c24f26d12c WatchSource:0}: Error finding container c84a545afdf1d3e3e0f8ae99f2e20bc193fc01572486611bbb15f3c24f26d12c: Status 404 returned error can't find the container with id c84a545afdf1d3e3e0f8ae99f2e20bc193fc01572486611bbb15f3c24f26d12c
Jul 14 00:59:32.117336 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.117308    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 00:59:32.154106 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:32.154066    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
Jul 14 00:59:32.225029 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.224953    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 00:59:32.244922 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.244887    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
Jul 14 00:59:32.257619 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.257587    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 00:59:32.258627 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.258612    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
Jul 14 00:59:32.266998 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.266951    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 00:59:32.826310 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.826269    2567 apiserver.go:52] "Watching apiserver"
Jul 14 00:59:32.833961 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.833605    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 14 00:59:32.834771 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.834730    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal","openshift-multus/multus-2mx5v","openshift-multus/network-metrics-daemon-jfg2f","kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6","openshift-cluster-node-tuning-operator/tuned-ggmtc","openshift-image-registry/node-ca-jpnj7","openshift-multus/multus-additional-cni-plugins-cg7w8","openshift-network-diagnostics/network-check-target-6dnxf","openshift-network-operator/iptables-alerter-hlqcw","openshift-ovn-kubernetes/ovnkube-node-f6kbd","kube-system/global-pull-secret-syncer-q8j4f","kube-system/konnectivity-agent-2z2tl","openshift-dns/node-resolver-d4fkp"]
Jul 14 00:59:32.836749 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.836716    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.839302 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.838793    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.839690 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.839610    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:32.839840 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:32.839682    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 00:59:32.839936 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.839886    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 14 00:59:32.840011 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.839967    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 14 00:59:32.840129 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.840112    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tr6kk\""
Jul 14 00:59:32.840708 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.840495    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 14 00:59:32.840708 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.840765    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 14 00:59:32.840708 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.840787    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 14 00:59:32.840708 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.840842    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 14 00:59:32.843469 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.841913    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9z4w8\""
Jul 14 00:59:32.846126 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.844384    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 14 00:59:32.846126 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.844613    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 14 00:59:32.846126 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.844655    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 14 00:59:32.846126 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.844841    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 14 00:59:32.846126 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.844940    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 14 00:59:32.846126 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.845685    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.846486 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.846427    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2z2tl"
Jul 14 00:59:32.849234 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.847466    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 14 00:59:32.849234 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.847690    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:32.849234 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.848474    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 14 00:59:32.849234 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.848575    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jpnj7"
Jul 14 00:59:32.849234 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.848745    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 14 00:59:32.849234 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.848952    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8nbln\""
Jul 14 00:59:32.849234 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.849038    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7gdfp\""
Jul 14 00:59:32.849633 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.849442    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 14 00:59:32.849633 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.849499    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 14 00:59:32.850450 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.850023    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6gcnj\""
Jul 14 00:59:32.850450 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.850190    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.855563    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.858836    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.860683    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-stz8p\""
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.860927    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-cnibin\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.860971    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-systemd-units\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861005    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-run-netns\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861060    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-run-openvswitch\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861105    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-var-lib-kubelet\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861140    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-kubelet\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861170    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-ovnkube-script-lib\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861202    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-run-netns\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861235    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-multus-conf-dir\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861277    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkz9\" (UniqueName: \"kubernetes.io/projected/f0ebac4e-8f5b-4820-b018-8386d03067f1-kube-api-access-mtkz9\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861308    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-slash\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861337    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-ovnkube-config\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861435    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2nbl\" (UniqueName: \"kubernetes.io/projected/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-kube-api-access-z2nbl\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861471    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861467    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a144e7e-6d83-49a5-90c8-6052b6734d80-cni-binary-copy\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.863504 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861736    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861769    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-multus-socket-dir-parent\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861805    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a144e7e-6d83-49a5-90c8-6052b6734d80-multus-daemon-config\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861832    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-etc-kubernetes\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861864    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/336250ce-5e32-48f2-88a0-365516e9b4b7-agent-certs\") pod \"konnectivity-agent-2z2tl\" (UID: \"336250ce-5e32-48f2-88a0-365516e9b4b7\") " pod="kube-system/konnectivity-agent-2z2tl"
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861898    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-var-lib-openvswitch\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861945    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-etc-openvswitch\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861976    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.861973    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-node-log\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862021    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-ovn-node-metrics-cert\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862074    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-system-cni-dir\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862107    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-var-lib-cni-bin\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862122    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862137    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k85wd\" (UniqueName: \"kubernetes.io/projected/3a144e7e-6d83-49a5-90c8-6052b6734d80-kube-api-access-k85wd\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862172    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-registration-dir\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862204    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-run-systemd\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862248    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-run-ovn\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862279    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-log-socket\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.864781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862309    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-cni-netd\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862348    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-env-overrides\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862374    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862383    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-run-multus-certs\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862415    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/336250ce-5e32-48f2-88a0-365516e9b4b7-konnectivity-ca\") pod \"konnectivity-agent-2z2tl\" (UID: \"336250ce-5e32-48f2-88a0-365516e9b4b7\") " pod="kube-system/konnectivity-agent-2z2tl"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862435    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862449    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-etc-selinux\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862482    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-run-ovn-kubernetes\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862507    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-cni-bin\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862526    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-multus-cni-dir\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:32.862524    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862545    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-os-release\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862376    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hlqcw"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862572    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-run-k8s-cni-cncf-io\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862593    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-hostroot\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862609    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862638    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-var-lib-cni-multus\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862674    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862695    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d4fkp"
Jul 14 00:59:32.865728 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862708    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-socket-dir\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.866662 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862749    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-device-dir\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.866662 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862777    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8phk\" (UniqueName: \"kubernetes.io/projected/bfc53be7-89a1-4e54-9188-a194c7a4c095-kube-api-access-j8phk\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.866662 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862825    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.866662 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.862860    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-sys-fs\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.866662 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.863072    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f27tz\""
Jul 14 00:59:32.866662 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.863154    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:32.866662 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:32.863207    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 00:59:32.868556 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.867625    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 14 00:59:32.868556 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.867732    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 14 00:59:32.868556 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.867923    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dzfzm\""
Jul 14 00:59:32.868556 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.867946    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mv7w7\""
Jul 14 00:59:32.868556 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.868075    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 14 00:59:32.868556 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.868186    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 14 00:59:32.868556 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.868270    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 14 00:59:32.871095 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.870966    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 00:59:32.878550 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.878492    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q9bwl"
Jul 14 00:59:32.886344 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.886313    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q9bwl"
Jul 14 00:59:32.945625 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.945591    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 14 00:59:32.963339 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963305    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a747595-65ad-41fe-a1f1-d0914d6eadc2-dbus\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:32.963533 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963348    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-slash\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.963533 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963384    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2nbl\" (UniqueName: \"kubernetes.io/projected/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-kube-api-access-z2nbl\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.963533 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963402    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a144e7e-6d83-49a5-90c8-6052b6734d80-cni-binary-copy\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.963533 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963425    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a144e7e-6d83-49a5-90c8-6052b6734d80-multus-daemon-config\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.963533 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963446    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-etc-kubernetes\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.963533 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963463    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-registration-dir\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.963533 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963479    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-lib-modules\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:32.963533 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963495    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-var-lib-openvswitch\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.963533 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963509    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-etc-openvswitch\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963542    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-node-log\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963560    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-var-lib-cni-bin\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963576    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k85wd\" (UniqueName: \"kubernetes.io/projected/3a144e7e-6d83-49a5-90c8-6052b6734d80-kube-api-access-k85wd\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963592    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-etc-selinux\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963607    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8phk\" (UniqueName: \"kubernetes.io/projected/bfc53be7-89a1-4e54-9188-a194c7a4c095-kube-api-access-j8phk\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963624    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a58647a6-7a3b-4298-b6b5-a18848141893-etc-tuned\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963641    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-log-socket\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963667    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-cni-netd\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963694    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-env-overrides\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963717    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/336250ce-5e32-48f2-88a0-365516e9b4b7-konnectivity-ca\") pod \"konnectivity-agent-2z2tl\" (UID: \"336250ce-5e32-48f2-88a0-365516e9b4b7\") " pod="kube-system/konnectivity-agent-2z2tl"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963733    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-run\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963748    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5459bba-8c2e-4be1-802f-a462a23327f6-host\") pod \"node-ca-jpnj7\" (UID: \"b5459bba-8c2e-4be1-802f-a462a23327f6\") " pod="openshift-image-registry/node-ca-jpnj7"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963766    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31b7f78f-0db2-469f-886d-a83f025c6a62-system-cni-dir\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963784    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/31b7f78f-0db2-469f-886d-a83f025c6a62-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963803    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-run-ovn-kubernetes\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963820    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-cni-bin\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.964111 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963849    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-hostroot\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963875    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963891    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-device-dir\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963909    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31b7f78f-0db2-469f-886d-a83f025c6a62-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963926    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-socket-dir\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963942    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-host\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963959    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5459bba-8c2e-4be1-802f-a462a23327f6-serviceca\") pod \"node-ca-jpnj7\" (UID: \"b5459bba-8c2e-4be1-802f-a462a23327f6\") " pod="openshift-image-registry/node-ca-jpnj7"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.963982    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d726e56-7cd0-49b6-9d1e-3949bdc47970-iptables-alerter-script\") pod \"iptables-alerter-hlqcw\" (UID: \"0d726e56-7cd0-49b6-9d1e-3949bdc47970\") " pod="openshift-network-operator/iptables-alerter-hlqcw"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964003    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d726e56-7cd0-49b6-9d1e-3949bdc47970-host-slash\") pod \"iptables-alerter-hlqcw\" (UID: \"0d726e56-7cd0-49b6-9d1e-3949bdc47970\") " pod="openshift-network-operator/iptables-alerter-hlqcw"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964050    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-sys-fs\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964073    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-kubernetes\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964094    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-var-lib-kubelet\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964110    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb9mw\" (UniqueName: \"kubernetes.io/projected/b5459bba-8c2e-4be1-802f-a462a23327f6-kube-api-access-tb9mw\") pod \"node-ca-jpnj7\" (UID: \"b5459bba-8c2e-4be1-802f-a462a23327f6\") " pod="openshift-image-registry/node-ca-jpnj7"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964127    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-cnibin\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964151    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-systemd-units\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964171    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-var-lib-kubelet\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964194    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a747595-65ad-41fe-a1f1-d0914d6eadc2-kubelet-config\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:32.964996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964240    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-kubelet\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964260    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-ovnkube-script-lib\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964277    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-run-netns\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964294    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-multus-conf-dir\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964311    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkz9\" (UniqueName: \"kubernetes.io/projected/f0ebac4e-8f5b-4820-b018-8386d03067f1-kube-api-access-mtkz9\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964335    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a58647a6-7a3b-4298-b6b5-a18848141893-tmp\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964356    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/31b7f78f-0db2-469f-886d-a83f025c6a62-os-release\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964380    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-ovnkube-config\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964399    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-multus-socket-dir-parent\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964423    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/336250ce-5e32-48f2-88a0-365516e9b4b7-agent-certs\") pod \"konnectivity-agent-2z2tl\" (UID: \"336250ce-5e32-48f2-88a0-365516e9b4b7\") " pod="kube-system/konnectivity-agent-2z2tl"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964451    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-sysctl-d\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964468    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-systemd\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964487    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31b7f78f-0db2-469f-886d-a83f025c6a62-cnibin\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964515    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/31b7f78f-0db2-469f-886d-a83f025c6a62-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964541    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964560    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-ovn-node-metrics-cert\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964584    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-system-cni-dir\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.965735 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964605    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9frgn\" (UniqueName: \"kubernetes.io/projected/2998116f-cba6-4d84-a68c-c4da115286ad-kube-api-access-9frgn\") pod \"node-resolver-d4fkp\" (UID: \"2998116f-cba6-4d84-a68c-c4da115286ad\") " pod="openshift-dns/node-resolver-d4fkp"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964628    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-run-systemd\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964653    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-run-ovn\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964680    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-run-multus-certs\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964705    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x84kt\" (UniqueName: \"kubernetes.io/projected/a58647a6-7a3b-4298-b6b5-a18848141893-kube-api-access-x84kt\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964727    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5nj\" (UniqueName: \"kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj\") pod \"network-check-target-6dnxf\" (UID: \"455ca3e1-0a9d-4b8e-b150-06ba86301ae5\") " pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964764    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2w4j\" (UniqueName: \"kubernetes.io/projected/0d726e56-7cd0-49b6-9d1e-3949bdc47970-kube-api-access-l2w4j\") pod \"iptables-alerter-hlqcw\" (UID: \"0d726e56-7cd0-49b6-9d1e-3949bdc47970\") " pod="openshift-network-operator/iptables-alerter-hlqcw"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964788    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-multus-cni-dir\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964816    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-os-release\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964838    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-run-k8s-cni-cncf-io\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964855    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-modprobe-d\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964875    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-sysconfig\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964907    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-var-lib-cni-multus\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964931    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964956    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.964985    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2998116f-cba6-4d84-a68c-c4da115286ad-tmp-dir\") pod \"node-resolver-d4fkp\" (UID: \"2998116f-cba6-4d84-a68c-c4da115286ad\") " pod="openshift-dns/node-resolver-d4fkp"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.965013    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31b7f78f-0db2-469f-886d-a83f025c6a62-cni-binary-copy\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:32.966454 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.965058    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6bf4\" (UniqueName: \"kubernetes.io/projected/31b7f78f-0db2-469f-886d-a83f025c6a62-kube-api-access-r6bf4\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:32.967022 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.965081    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-run-netns\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.967022 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.965106    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-run-openvswitch\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.967022 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.965126    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2998116f-cba6-4d84-a68c-c4da115286ad-hosts-file\") pod \"node-resolver-d4fkp\" (UID: \"2998116f-cba6-4d84-a68c-c4da115286ad\") " pod="openshift-dns/node-resolver-d4fkp"
Jul 14 00:59:32.967022 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.965142    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-sysctl-conf\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:32.967022 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.965159    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-sys\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:32.967022 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.965261    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-slash\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.967022 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.966164    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-run-ovn-kubernetes\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.967022 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.966315    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-hostroot\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.967022 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.966328    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-run-netns\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.967022 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.966366    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-multus-conf-dir\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.967022 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.966849    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a144e7e-6d83-49a5-90c8-6052b6734d80-cni-binary-copy\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.967022 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.966943    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-run-multus-certs\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.967022 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.966961    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-ovnkube-script-lib\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.967679 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967065    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-ovnkube-config\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.967679 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967073    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-registration-dir\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.967679 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967196    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-multus-socket-dir-parent\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.967679 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967202    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-multus-cni-dir\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.967679 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967273    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-os-release\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.967679 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967328    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-run-systemd\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.967679 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967334    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-var-lib-cni-multus\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.967679 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967441    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-socket-dir\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.967679 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967507    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-sys-fs\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.967679 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967559    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-cnibin\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.967679 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967589    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 14 00:59:32.967679 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967626    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-systemd-units\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.967679 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:32.967599    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:59:32.967679 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967646    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.968345 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967713    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-run-netns\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.968345 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:32.967728    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs podName:f0ebac4e-8f5b-4820-b018-8386d03067f1 nodeName:}" failed. No retries permitted until 2026-07-14 00:59:33.46768646 +0000 UTC m=+3.032095514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs") pod "network-metrics-daemon-jfg2f" (UID: "f0ebac4e-8f5b-4820-b018-8386d03067f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:59:32.968345 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967751    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-run-openvswitch\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.968345 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967985    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-cni-netd\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.968345 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968022    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-run-ovn\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.968345 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968275    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-run-k8s-cni-cncf-io\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.968534 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968443    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-env-overrides\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.968534 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968459    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-var-lib-kubelet\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.968534 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968446    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-var-lib-openvswitch\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.968534 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968492    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.968534 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968512    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-node-log\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.968534 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968520    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-etc-kubernetes\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.968713 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968557    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-etc-openvswitch\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.968713 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968569    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-etc-selinux\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.968713 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968577    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-host-var-lib-cni-bin\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.968713 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968587    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-log-socket\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.968713 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968608    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfc53be7-89a1-4e54-9188-a194c7a4c095-device-dir\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.968713 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968616    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-kubelet\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.968713 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.967596    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-host-cni-bin\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.968713 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968637    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a144e7e-6d83-49a5-90c8-6052b6734d80-system-cni-dir\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.968991 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.968973    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/336250ce-5e32-48f2-88a0-365516e9b4b7-konnectivity-ca\") pod \"konnectivity-agent-2z2tl\" (UID: \"336250ce-5e32-48f2-88a0-365516e9b4b7\") " pod="kube-system/konnectivity-agent-2z2tl"
Jul 14 00:59:32.970077 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.970030    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a144e7e-6d83-49a5-90c8-6052b6734d80-multus-daemon-config\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:32.972814 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.972788    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-ovn-node-metrics-cert\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.972995 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.972906    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/336250ce-5e32-48f2-88a0-365516e9b4b7-agent-certs\") pod \"konnectivity-agent-2z2tl\" (UID: \"336250ce-5e32-48f2-88a0-365516e9b4b7\") " pod="kube-system/konnectivity-agent-2z2tl"
Jul 14 00:59:32.976276 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.974817    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal" event={"ID":"9105b408cddaedf77cc3bc80f328aa9e","Type":"ContainerStarted","Data":"c84a545afdf1d3e3e0f8ae99f2e20bc193fc01572486611bbb15f3c24f26d12c"}
Jul 14 00:59:32.976276 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.976224    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal" event={"ID":"00280b56ea67b292319742fa9e3451f2","Type":"ContainerStarted","Data":"17c32e6723b809c7d8d97f7c70286c3c664701a2865dfe2844a1400ea9bcb120"}
Jul 14 00:59:32.981014 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.980967    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2nbl\" (UniqueName: \"kubernetes.io/projected/ca6d2ac7-b806-485c-b968-f1e2bb8a1a45-kube-api-access-z2nbl\") pod \"ovnkube-node-f6kbd\" (UID: \"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:32.981556 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.981526    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8phk\" (UniqueName: \"kubernetes.io/projected/bfc53be7-89a1-4e54-9188-a194c7a4c095-kube-api-access-j8phk\") pod \"aws-ebs-csi-driver-node-2v4f6\" (UID: \"bfc53be7-89a1-4e54-9188-a194c7a4c095\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:32.982070 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.982026    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkz9\" (UniqueName: \"kubernetes.io/projected/f0ebac4e-8f5b-4820-b018-8386d03067f1-kube-api-access-mtkz9\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:32.982666 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:32.982644    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k85wd\" (UniqueName: \"kubernetes.io/projected/3a144e7e-6d83-49a5-90c8-6052b6734d80-kube-api-access-k85wd\") pod \"multus-2mx5v\" (UID: \"3a144e7e-6d83-49a5-90c8-6052b6734d80\") " pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:33.003246 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.003210    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 00:59:33.065567 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065530    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31b7f78f-0db2-469f-886d-a83f025c6a62-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.065778 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065579    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-host\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.065778 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065609    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5459bba-8c2e-4be1-802f-a462a23327f6-serviceca\") pod \"node-ca-jpnj7\" (UID: \"b5459bba-8c2e-4be1-802f-a462a23327f6\") " pod="openshift-image-registry/node-ca-jpnj7"
Jul 14 00:59:33.065778 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065636    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d726e56-7cd0-49b6-9d1e-3949bdc47970-iptables-alerter-script\") pod \"iptables-alerter-hlqcw\" (UID: \"0d726e56-7cd0-49b6-9d1e-3949bdc47970\") " pod="openshift-network-operator/iptables-alerter-hlqcw"
Jul 14 00:59:33.065778 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065662    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d726e56-7cd0-49b6-9d1e-3949bdc47970-host-slash\") pod \"iptables-alerter-hlqcw\" (UID: \"0d726e56-7cd0-49b6-9d1e-3949bdc47970\") " pod="openshift-network-operator/iptables-alerter-hlqcw"
Jul 14 00:59:33.065778 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065686    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-kubernetes\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.065778 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065709    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-var-lib-kubelet\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.065778 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065733    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb9mw\" (UniqueName: \"kubernetes.io/projected/b5459bba-8c2e-4be1-802f-a462a23327f6-kube-api-access-tb9mw\") pod \"node-ca-jpnj7\" (UID: \"b5459bba-8c2e-4be1-802f-a462a23327f6\") " pod="openshift-image-registry/node-ca-jpnj7"
Jul 14 00:59:33.065778 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065769    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a747595-65ad-41fe-a1f1-d0914d6eadc2-kubelet-config\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065798    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a58647a6-7a3b-4298-b6b5-a18848141893-tmp\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065822    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/31b7f78f-0db2-469f-886d-a83f025c6a62-os-release\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065850    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-sysctl-d\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065875    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-systemd\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065898    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31b7f78f-0db2-469f-886d-a83f025c6a62-cnibin\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065923    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/31b7f78f-0db2-469f-886d-a83f025c6a62-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065949    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.065977    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9frgn\" (UniqueName: \"kubernetes.io/projected/2998116f-cba6-4d84-a68c-c4da115286ad-kube-api-access-9frgn\") pod \"node-resolver-d4fkp\" (UID: \"2998116f-cba6-4d84-a68c-c4da115286ad\") " pod="openshift-dns/node-resolver-d4fkp"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066005    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x84kt\" (UniqueName: \"kubernetes.io/projected/a58647a6-7a3b-4298-b6b5-a18848141893-kube-api-access-x84kt\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066054    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5nj\" (UniqueName: \"kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj\") pod \"network-check-target-6dnxf\" (UID: \"455ca3e1-0a9d-4b8e-b150-06ba86301ae5\") " pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066079    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2w4j\" (UniqueName: \"kubernetes.io/projected/0d726e56-7cd0-49b6-9d1e-3949bdc47970-kube-api-access-l2w4j\") pod \"iptables-alerter-hlqcw\" (UID: \"0d726e56-7cd0-49b6-9d1e-3949bdc47970\") " pod="openshift-network-operator/iptables-alerter-hlqcw"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066109    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-modprobe-d\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066133    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-sysconfig\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066174    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2998116f-cba6-4d84-a68c-c4da115286ad-tmp-dir\") pod \"node-resolver-d4fkp\" (UID: \"2998116f-cba6-4d84-a68c-c4da115286ad\") " pod="openshift-dns/node-resolver-d4fkp"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066199    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31b7f78f-0db2-469f-886d-a83f025c6a62-cni-binary-copy\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066225    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6bf4\" (UniqueName: \"kubernetes.io/projected/31b7f78f-0db2-469f-886d-a83f025c6a62-kube-api-access-r6bf4\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.066314 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066253    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2998116f-cba6-4d84-a68c-c4da115286ad-hosts-file\") pod \"node-resolver-d4fkp\" (UID: \"2998116f-cba6-4d84-a68c-c4da115286ad\") " pod="openshift-dns/node-resolver-d4fkp"
Jul 14 00:59:33.067194 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066278    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-sysctl-conf\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.067194 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066300    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-sys\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.067194 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066326    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a747595-65ad-41fe-a1f1-d0914d6eadc2-dbus\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:33.067194 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066369    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-lib-modules\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.067194 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066404    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a58647a6-7a3b-4298-b6b5-a18848141893-etc-tuned\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.067194 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066433    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-run\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.067194 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066460    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5459bba-8c2e-4be1-802f-a462a23327f6-host\") pod \"node-ca-jpnj7\" (UID: \"b5459bba-8c2e-4be1-802f-a462a23327f6\") " pod="openshift-image-registry/node-ca-jpnj7"
Jul 14 00:59:33.067194 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066486    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31b7f78f-0db2-469f-886d-a83f025c6a62-system-cni-dir\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.067194 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.066510    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/31b7f78f-0db2-469f-886d-a83f025c6a62-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.067194 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067045    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/31b7f78f-0db2-469f-886d-a83f025c6a62-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.067194 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067139    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a747595-65ad-41fe-a1f1-d0914d6eadc2-kubelet-config\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:33.067194 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067183    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31b7f78f-0db2-469f-886d-a83f025c6a62-cnibin\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.067194 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067182    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-host\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067199    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-sysconfig\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067199    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-systemd\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067280    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/31b7f78f-0db2-469f-886d-a83f025c6a62-os-release\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.067287    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.067355    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret podName:1a747595-65ad-41fe-a1f1-d0914d6eadc2 nodeName:}" failed. No retries permitted until 2026-07-14 00:59:33.567337224 +0000 UTC m=+3.131746262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret") pod "global-pull-secret-syncer-q8j4f" (UID: "1a747595-65ad-41fe-a1f1-d0914d6eadc2") : object "kube-system"/"original-pull-secret" not registered
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067358    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31b7f78f-0db2-469f-886d-a83f025c6a62-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067411    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-sysctl-d\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067462    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-sysctl-conf\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067508    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-modprobe-d\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067517    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-sys\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067566    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-run\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067606    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31b7f78f-0db2-469f-886d-a83f025c6a62-system-cni-dir\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067640    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-var-lib-kubelet\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067648    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2998116f-cba6-4d84-a68c-c4da115286ad-tmp-dir\") pod \"node-resolver-d4fkp\" (UID: \"2998116f-cba6-4d84-a68c-c4da115286ad\") " pod="openshift-dns/node-resolver-d4fkp"
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067665    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a747595-65ad-41fe-a1f1-d0914d6eadc2-dbus\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067182    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2998116f-cba6-4d84-a68c-c4da115286ad-hosts-file\") pod \"node-resolver-d4fkp\" (UID: \"2998116f-cba6-4d84-a68c-c4da115286ad\") " pod="openshift-dns/node-resolver-d4fkp"
Jul 14 00:59:33.067798 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067763    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-lib-modules\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.068945 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067910    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a58647a6-7a3b-4298-b6b5-a18848141893-etc-kubernetes\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.068945 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067954    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5459bba-8c2e-4be1-802f-a462a23327f6-host\") pod \"node-ca-jpnj7\" (UID: \"b5459bba-8c2e-4be1-802f-a462a23327f6\") " pod="openshift-image-registry/node-ca-jpnj7"
Jul 14 00:59:33.068945 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.067959    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5459bba-8c2e-4be1-802f-a462a23327f6-serviceca\") pod \"node-ca-jpnj7\" (UID: \"b5459bba-8c2e-4be1-802f-a462a23327f6\") " pod="openshift-image-registry/node-ca-jpnj7"
Jul 14 00:59:33.068945 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.068011    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d726e56-7cd0-49b6-9d1e-3949bdc47970-host-slash\") pod \"iptables-alerter-hlqcw\" (UID: \"0d726e56-7cd0-49b6-9d1e-3949bdc47970\") " pod="openshift-network-operator/iptables-alerter-hlqcw"
Jul 14 00:59:33.068945 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.068258    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0d726e56-7cd0-49b6-9d1e-3949bdc47970-iptables-alerter-script\") pod \"iptables-alerter-hlqcw\" (UID: \"0d726e56-7cd0-49b6-9d1e-3949bdc47970\") " pod="openshift-network-operator/iptables-alerter-hlqcw"
Jul 14 00:59:33.068945 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.068342    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31b7f78f-0db2-469f-886d-a83f025c6a62-cni-binary-copy\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.068945 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.068447    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/31b7f78f-0db2-469f-886d-a83f025c6a62-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.070167 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.070108    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a58647a6-7a3b-4298-b6b5-a18848141893-tmp\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.073376 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.073347    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 00:59:33.073376 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.073378    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 00:59:33.073541 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.073397    2567 projected.go:196] Error preparing data for projected volume kube-api-access-gg5nj for pod openshift-network-diagnostics/network-check-target-6dnxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:59:33.073541 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.073476    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj podName:455ca3e1-0a9d-4b8e-b150-06ba86301ae5 nodeName:}" failed. No retries permitted until 2026-07-14 00:59:33.573459874 +0000 UTC m=+3.137868919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gg5nj" (UniqueName: "kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj") pod "network-check-target-6dnxf" (UID: "455ca3e1-0a9d-4b8e-b150-06ba86301ae5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:59:33.075672 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.075639    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a58647a6-7a3b-4298-b6b5-a18848141893-etc-tuned\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.079047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.078981    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x84kt\" (UniqueName: \"kubernetes.io/projected/a58647a6-7a3b-4298-b6b5-a18848141893-kube-api-access-x84kt\") pod \"tuned-ggmtc\" (UID: \"a58647a6-7a3b-4298-b6b5-a18848141893\") " pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.079146 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.079085    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9frgn\" (UniqueName: \"kubernetes.io/projected/2998116f-cba6-4d84-a68c-c4da115286ad-kube-api-access-9frgn\") pod \"node-resolver-d4fkp\" (UID: \"2998116f-cba6-4d84-a68c-c4da115286ad\") " pod="openshift-dns/node-resolver-d4fkp"
Jul 14 00:59:33.079496 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.079477    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2w4j\" (UniqueName: \"kubernetes.io/projected/0d726e56-7cd0-49b6-9d1e-3949bdc47970-kube-api-access-l2w4j\") pod \"iptables-alerter-hlqcw\" (UID: \"0d726e56-7cd0-49b6-9d1e-3949bdc47970\") " pod="openshift-network-operator/iptables-alerter-hlqcw"
Jul 14 00:59:33.079676 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.079658    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb9mw\" (UniqueName: \"kubernetes.io/projected/b5459bba-8c2e-4be1-802f-a462a23327f6-kube-api-access-tb9mw\") pod \"node-ca-jpnj7\" (UID: \"b5459bba-8c2e-4be1-802f-a462a23327f6\") " pod="openshift-image-registry/node-ca-jpnj7"
Jul 14 00:59:33.081453 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.081428    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6bf4\" (UniqueName: \"kubernetes.io/projected/31b7f78f-0db2-469f-886d-a83f025c6a62-kube-api-access-r6bf4\") pod \"multus-additional-cni-plugins-cg7w8\" (UID: \"31b7f78f-0db2-469f-886d-a83f025c6a62\") " pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.167680 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.167635    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:33.178327 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.178293    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jpnj7"
Jul 14 00:59:33.187577 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.187540    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2mx5v"
Jul 14 00:59:33.192013 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.191985    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6"
Jul 14 00:59:33.201559 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.201511    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2z2tl"
Jul 14 00:59:33.209944 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.209905    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ggmtc"
Jul 14 00:59:33.217189 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.217162    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cg7w8"
Jul 14 00:59:33.226349 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.226317    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hlqcw"
Jul 14 00:59:33.231600 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.231368    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d4fkp"
Jul 14 00:59:33.469880 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.469845    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:33.470082 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.469970    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:59:33.470132 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.470115    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs podName:f0ebac4e-8f5b-4820-b018-8386d03067f1 nodeName:}" failed. No retries permitted until 2026-07-14 00:59:34.470098713 +0000 UTC m=+4.034507748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs") pod "network-metrics-daemon-jfg2f" (UID: "f0ebac4e-8f5b-4820-b018-8386d03067f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:59:33.570558 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.570507    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:33.570748 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.570626    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 00:59:33.570748 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.570710    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret podName:1a747595-65ad-41fe-a1f1-d0914d6eadc2 nodeName:}" failed. No retries permitted until 2026-07-14 00:59:34.570688356 +0000 UTC m=+4.135097393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret") pod "global-pull-secret-syncer-q8j4f" (UID: "1a747595-65ad-41fe-a1f1-d0914d6eadc2") : object "kube-system"/"original-pull-secret" not registered
Jul 14 00:59:33.671040 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.670970    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5nj\" (UniqueName: \"kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj\") pod \"network-check-target-6dnxf\" (UID: \"455ca3e1-0a9d-4b8e-b150-06ba86301ae5\") " pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:33.671256 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.671182    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 00:59:33.671256 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.671202    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 00:59:33.671256 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.671213    2567 projected.go:196] Error preparing data for projected volume kube-api-access-gg5nj for pod openshift-network-diagnostics/network-check-target-6dnxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:59:33.671442 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.671269    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj podName:455ca3e1-0a9d-4b8e-b150-06ba86301ae5 nodeName:}" failed. No retries permitted until 2026-07-14 00:59:34.671254962 +0000 UTC m=+4.235663996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gg5nj" (UniqueName: "kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj") pod "network-check-target-6dnxf" (UID: "455ca3e1-0a9d-4b8e-b150-06ba86301ae5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:59:33.889143 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.889077    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 00:54:33 +0000 UTC" deadline="2028-01-22 23:17:04.796634041 +0000 UTC"
Jul 14 00:59:33.889143 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.889144    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13390h17m30.907499104s"
Jul 14 00:59:33.968982 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.968920    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:33.969167 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.969079    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 00:59:33.969167 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:33.969089    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:33.969272 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:33.969238    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 00:59:34.211086 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:34.211047    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a144e7e_6d83_49a5_90c8_6052b6734d80.slice/crio-776db9ed02dfb4c115aa69067111afb27f480db0e4e515c3963b7d4d25db34bc WatchSource:0}: Error finding container 776db9ed02dfb4c115aa69067111afb27f480db0e4e515c3963b7d4d25db34bc: Status 404 returned error can't find the container with id 776db9ed02dfb4c115aa69067111afb27f480db0e4e515c3963b7d4d25db34bc
Jul 14 00:59:34.217816 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:34.217784    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2998116f_cba6_4d84_a68c_c4da115286ad.slice/crio-38872809aefbeea65b74f9eada4958b357f2148cf3787cdcec7760c7c008fa2e WatchSource:0}: Error finding container 38872809aefbeea65b74f9eada4958b357f2148cf3787cdcec7760c7c008fa2e: Status 404 returned error can't find the container with id 38872809aefbeea65b74f9eada4958b357f2148cf3787cdcec7760c7c008fa2e
Jul 14 00:59:34.219297 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:34.219274    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58647a6_7a3b_4298_b6b5_a18848141893.slice/crio-52e206617bf4a77394423c871778a8ce26402c4363109fc3942cf4721e673516 WatchSource:0}: Error finding container 52e206617bf4a77394423c871778a8ce26402c4363109fc3942cf4721e673516: Status 404 returned error can't find the container with id 52e206617bf4a77394423c871778a8ce26402c4363109fc3942cf4721e673516
Jul 14 00:59:34.220627 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:34.220605    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6d2ac7_b806_485c_b968_f1e2bb8a1a45.slice/crio-b2a7109287de21818a69a86e338e44a2fda7d3abe0c0b178071ac7fd80d792b6 WatchSource:0}: Error finding container b2a7109287de21818a69a86e338e44a2fda7d3abe0c0b178071ac7fd80d792b6: Status 404 returned error can't find the container with id b2a7109287de21818a69a86e338e44a2fda7d3abe0c0b178071ac7fd80d792b6
Jul 14 00:59:34.224465 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:34.224415    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d726e56_7cd0_49b6_9d1e_3949bdc47970.slice/crio-bdbe68f454bc20c4af4a8a8d591af926768dda71eb2680525cf78aa54c1bca66 WatchSource:0}: Error finding container bdbe68f454bc20c4af4a8a8d591af926768dda71eb2680525cf78aa54c1bca66: Status 404 returned error can't find the container with id bdbe68f454bc20c4af4a8a8d591af926768dda71eb2680525cf78aa54c1bca66
Jul 14 00:59:34.225845 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:34.225805    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b7f78f_0db2_469f_886d_a83f025c6a62.slice/crio-4ab32682625f496c02f845cd63210f52fafd51008559f81aa7286fbcbc089459 WatchSource:0}: Error finding container 4ab32682625f496c02f845cd63210f52fafd51008559f81aa7286fbcbc089459: Status 404 returned error can't find the container with id 4ab32682625f496c02f845cd63210f52fafd51008559f81aa7286fbcbc089459
Jul 14 00:59:34.227130 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:34.227105    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336250ce_5e32_48f2_88a0_365516e9b4b7.slice/crio-24e3f5b20ee0147bdd485840fadc4e5b640881d5ef9f503c8e494b7a8ad446f9 WatchSource:0}: Error finding container 24e3f5b20ee0147bdd485840fadc4e5b640881d5ef9f503c8e494b7a8ad446f9: Status 404 returned error can't find the container with id 24e3f5b20ee0147bdd485840fadc4e5b640881d5ef9f503c8e494b7a8ad446f9
Jul 14 00:59:34.227925 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:34.227850    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5459bba_8c2e_4be1_802f_a462a23327f6.slice/crio-bf82e13de58e9d3c39da8a364e5a5873bb9a1bc272c32105f26dcac49449bfc7 WatchSource:0}: Error finding container bf82e13de58e9d3c39da8a364e5a5873bb9a1bc272c32105f26dcac49449bfc7: Status 404 returned error can't find the container with id bf82e13de58e9d3c39da8a364e5a5873bb9a1bc272c32105f26dcac49449bfc7
Jul 14 00:59:34.232687 ip-10-0-129-117 kubenswrapper[2567]: W0714 00:59:34.230262    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc53be7_89a1_4e54_9188_a194c7a4c095.slice/crio-0e72625e7d235b66e5d938d7f6a8ce6c94552e82f49f8f5a65bf75c7ef2904b1 WatchSource:0}: Error finding container 0e72625e7d235b66e5d938d7f6a8ce6c94552e82f49f8f5a65bf75c7ef2904b1: Status 404 returned error can't find the container with id 0e72625e7d235b66e5d938d7f6a8ce6c94552e82f49f8f5a65bf75c7ef2904b1
Jul 14 00:59:34.479737 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:34.479552    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:34.479861 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:34.479736    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:59:34.479861 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:34.479818    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs podName:f0ebac4e-8f5b-4820-b018-8386d03067f1 nodeName:}" failed. No retries permitted until 2026-07-14 00:59:36.479800474 +0000 UTC m=+6.044209509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs") pod "network-metrics-daemon-jfg2f" (UID: "f0ebac4e-8f5b-4820-b018-8386d03067f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:59:34.581237 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:34.581186    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:34.581395 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:34.581356    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 00:59:34.581465 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:34.581446    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret podName:1a747595-65ad-41fe-a1f1-d0914d6eadc2 nodeName:}" failed. No retries permitted until 2026-07-14 00:59:36.581423149 +0000 UTC m=+6.145832197 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret") pod "global-pull-secret-syncer-q8j4f" (UID: "1a747595-65ad-41fe-a1f1-d0914d6eadc2") : object "kube-system"/"original-pull-secret" not registered
Jul 14 00:59:34.681630 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:34.681579    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5nj\" (UniqueName: \"kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj\") pod \"network-check-target-6dnxf\" (UID: \"455ca3e1-0a9d-4b8e-b150-06ba86301ae5\") " pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:34.681848 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:34.681827    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 00:59:34.681924 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:34.681857    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 00:59:34.681924 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:34.681870    2567 projected.go:196] Error preparing data for projected volume kube-api-access-gg5nj for pod openshift-network-diagnostics/network-check-target-6dnxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:59:34.682058 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:34.681932    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj podName:455ca3e1-0a9d-4b8e-b150-06ba86301ae5 nodeName:}" failed. No retries permitted until 2026-07-14 00:59:36.681913088 +0000 UTC m=+6.246322126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gg5nj" (UniqueName: "kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj") pod "network-check-target-6dnxf" (UID: "455ca3e1-0a9d-4b8e-b150-06ba86301ae5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:59:34.890400 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:34.890210    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 00:54:33 +0000 UTC" deadline="2028-03-26 11:01:15.178089736 +0000 UTC"
Jul 14 00:59:34.890400 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:34.890277    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14914h1m40.287817267s"
Jul 14 00:59:34.989593 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:34.972173    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:34.990987 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:34.990561    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 00:59:34.997564 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:34.997483    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hlqcw" event={"ID":"0d726e56-7cd0-49b6-9d1e-3949bdc47970","Type":"ContainerStarted","Data":"bdbe68f454bc20c4af4a8a8d591af926768dda71eb2680525cf78aa54c1bca66"}
Jul 14 00:59:35.006271 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:35.006224    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2mx5v" event={"ID":"3a144e7e-6d83-49a5-90c8-6052b6734d80","Type":"ContainerStarted","Data":"776db9ed02dfb4c115aa69067111afb27f480db0e4e515c3963b7d4d25db34bc"}
Jul 14 00:59:35.018478 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:35.018432    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal" event={"ID":"9105b408cddaedf77cc3bc80f328aa9e","Type":"ContainerStarted","Data":"fbc51cd4b113a5f53bb7ff0b3f503ce7585220c8383036166cfff1cbc1435992"}
Jul 14 00:59:35.033095 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:35.033053    2567 generic.go:356] "Generic (PLEG): container finished" podID="00280b56ea67b292319742fa9e3451f2" containerID="d1b5254d1b068ff70d3c1567ca7a56223fe7771289cd6958b96bee711eae27f5" exitCode=0
Jul 14 00:59:35.033301 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:35.033187    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal" event={"ID":"00280b56ea67b292319742fa9e3451f2","Type":"ContainerDied","Data":"d1b5254d1b068ff70d3c1567ca7a56223fe7771289cd6958b96bee711eae27f5"}
Jul 14 00:59:35.035150 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:35.035098    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal" podStartSLOduration=3.03507766 podStartE2EDuration="3.03507766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:59:35.034122861 +0000 UTC m=+4.598531922" watchObservedRunningTime="2026-07-14 00:59:35.03507766 +0000 UTC m=+4.599486718"
Jul 14 00:59:35.049539 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:35.049456    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jpnj7" event={"ID":"b5459bba-8c2e-4be1-802f-a462a23327f6","Type":"ContainerStarted","Data":"bf82e13de58e9d3c39da8a364e5a5873bb9a1bc272c32105f26dcac49449bfc7"}
Jul 14 00:59:35.061500 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:35.061455    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd" event={"ID":"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45","Type":"ContainerStarted","Data":"b2a7109287de21818a69a86e338e44a2fda7d3abe0c0b178071ac7fd80d792b6"}
Jul 14 00:59:35.072162 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:35.072113    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ggmtc" event={"ID":"a58647a6-7a3b-4298-b6b5-a18848141893","Type":"ContainerStarted","Data":"52e206617bf4a77394423c871778a8ce26402c4363109fc3942cf4721e673516"}
Jul 14 00:59:35.081238 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:35.081191    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d4fkp" event={"ID":"2998116f-cba6-4d84-a68c-c4da115286ad","Type":"ContainerStarted","Data":"38872809aefbeea65b74f9eada4958b357f2148cf3787cdcec7760c7c008fa2e"}
Jul 14 00:59:35.091087 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:35.091040    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6" event={"ID":"bfc53be7-89a1-4e54-9188-a194c7a4c095","Type":"ContainerStarted","Data":"0e72625e7d235b66e5d938d7f6a8ce6c94552e82f49f8f5a65bf75c7ef2904b1"}
Jul 14 00:59:35.098977 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:35.098914    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2z2tl" event={"ID":"336250ce-5e32-48f2-88a0-365516e9b4b7","Type":"ContainerStarted","Data":"24e3f5b20ee0147bdd485840fadc4e5b640881d5ef9f503c8e494b7a8ad446f9"}
Jul 14 00:59:35.117918 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:35.117864    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cg7w8" event={"ID":"31b7f78f-0db2-469f-886d-a83f025c6a62","Type":"ContainerStarted","Data":"4ab32682625f496c02f845cd63210f52fafd51008559f81aa7286fbcbc089459"}
Jul 14 00:59:35.968588 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:35.968543    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:35.969179 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:35.968699    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 00:59:35.969179 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:35.969152    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:35.969437 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:35.969257    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 00:59:36.129519 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:36.129441    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal" event={"ID":"00280b56ea67b292319742fa9e3451f2","Type":"ContainerStarted","Data":"571b71edd3be9c07fc71e7c8d027967b43005316c1c22db9a01e034332a3d455"}
Jul 14 00:59:36.146934 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:36.146798    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal" podStartSLOduration=4.146771405 podStartE2EDuration="4.146771405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:59:36.145695682 +0000 UTC m=+5.710104742" watchObservedRunningTime="2026-07-14 00:59:36.146771405 +0000 UTC m=+5.711180470"
Jul 14 00:59:36.503165 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:36.503121    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:36.503385 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:36.503333    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:59:36.503452 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:36.503402    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs podName:f0ebac4e-8f5b-4820-b018-8386d03067f1 nodeName:}" failed. No retries permitted until 2026-07-14 00:59:40.503382732 +0000 UTC m=+10.067791770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs") pod "network-metrics-daemon-jfg2f" (UID: "f0ebac4e-8f5b-4820-b018-8386d03067f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:59:36.604521 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:36.604477    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:36.604736 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:36.604662    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 00:59:36.604736 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:36.604732    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret podName:1a747595-65ad-41fe-a1f1-d0914d6eadc2 nodeName:}" failed. No retries permitted until 2026-07-14 00:59:40.604710054 +0000 UTC m=+10.169119091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret") pod "global-pull-secret-syncer-q8j4f" (UID: "1a747595-65ad-41fe-a1f1-d0914d6eadc2") : object "kube-system"/"original-pull-secret" not registered
Jul 14 00:59:36.705820 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:36.705773    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5nj\" (UniqueName: \"kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj\") pod \"network-check-target-6dnxf\" (UID: \"455ca3e1-0a9d-4b8e-b150-06ba86301ae5\") " pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:36.706075 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:36.705987    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 00:59:36.706075 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:36.706037    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 00:59:36.706275 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:36.706262    2567 projected.go:196] Error preparing data for projected volume kube-api-access-gg5nj for pod openshift-network-diagnostics/network-check-target-6dnxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:59:36.706421 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:36.706403    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj podName:455ca3e1-0a9d-4b8e-b150-06ba86301ae5 nodeName:}" failed. No retries permitted until 2026-07-14 00:59:40.70632874 +0000 UTC m=+10.270737793 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gg5nj" (UniqueName: "kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj") pod "network-check-target-6dnxf" (UID: "455ca3e1-0a9d-4b8e-b150-06ba86301ae5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:59:36.971775 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:36.971739    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:36.972427 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:36.971887    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 00:59:37.970200 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:37.969568    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:37.970200 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:37.969681    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 00:59:37.970200 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:37.970042    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:37.970200 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:37.970159    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 00:59:38.971888 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:38.971852    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:38.972403 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:38.971997    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 00:59:39.969392 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:39.969353    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:39.969552 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:39.969522    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 00:59:39.969981 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:39.969947    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:39.970108 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:39.970084    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 00:59:40.540546 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:40.540504    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:40.541105 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:40.540677    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:59:40.541105 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:40.540747    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs podName:f0ebac4e-8f5b-4820-b018-8386d03067f1 nodeName:}" failed. No retries permitted until 2026-07-14 00:59:48.540725218 +0000 UTC m=+18.105134254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs") pod "network-metrics-daemon-jfg2f" (UID: "f0ebac4e-8f5b-4820-b018-8386d03067f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:59:40.641707 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:40.641662    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:40.641919 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:40.641798    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 00:59:40.641919 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:40.641883    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret podName:1a747595-65ad-41fe-a1f1-d0914d6eadc2 nodeName:}" failed. No retries permitted until 2026-07-14 00:59:48.641860809 +0000 UTC m=+18.206269855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret") pod "global-pull-secret-syncer-q8j4f" (UID: "1a747595-65ad-41fe-a1f1-d0914d6eadc2") : object "kube-system"/"original-pull-secret" not registered
Jul 14 00:59:40.743047 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:40.743002    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5nj\" (UniqueName: \"kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj\") pod \"network-check-target-6dnxf\" (UID: \"455ca3e1-0a9d-4b8e-b150-06ba86301ae5\") " pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:40.743263 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:40.743213    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 00:59:40.743263 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:40.743238    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 00:59:40.743263 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:40.743251    2567 projected.go:196] Error preparing data for projected volume kube-api-access-gg5nj for pod openshift-network-diagnostics/network-check-target-6dnxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:59:40.743512 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:40.743315    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj podName:455ca3e1-0a9d-4b8e-b150-06ba86301ae5 nodeName:}" failed. No retries permitted until 2026-07-14 00:59:48.743295183 +0000 UTC m=+18.307704220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gg5nj" (UniqueName: "kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj") pod "network-check-target-6dnxf" (UID: "455ca3e1-0a9d-4b8e-b150-06ba86301ae5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:59:40.969590 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:40.969544    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:40.969861 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:40.969647    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 00:59:41.969051 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:41.969009    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:41.969491 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:41.969146    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 00:59:41.969537 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:41.969509    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:41.969614 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:41.969595    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 00:59:42.969377 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:42.969327    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:42.969821 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:42.969464    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 00:59:43.968932 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:43.968893    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:43.969128 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:43.969060    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 00:59:43.969512 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:43.969483    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:43.969912 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:43.969586    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 00:59:44.969202 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:44.969170    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:44.969390 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:44.969296    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 00:59:45.969074 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:45.968980    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:45.969074 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:45.969007    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:45.969614 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:45.969136    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 00:59:45.969614 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:45.969272    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 00:59:46.968967 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:46.968927    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:46.969142 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:46.969065    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 00:59:47.968516 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:47.968475    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:47.968781 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:47.968528    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:47.968781 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:47.968632    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 00:59:47.968781 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:47.968764    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 00:59:48.605996 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:48.605946    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:48.606567 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:48.606110    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:59:48.606567 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:48.606170    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs podName:f0ebac4e-8f5b-4820-b018-8386d03067f1 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:04.606154026 +0000 UTC m=+34.170563061 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs") pod "network-metrics-daemon-jfg2f" (UID: "f0ebac4e-8f5b-4820-b018-8386d03067f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:59:48.707178 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:48.707144    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:48.707344 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:48.707307    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 00:59:48.707399 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:48.707380    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret podName:1a747595-65ad-41fe-a1f1-d0914d6eadc2 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:04.707358517 +0000 UTC m=+34.271767550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret") pod "global-pull-secret-syncer-q8j4f" (UID: "1a747595-65ad-41fe-a1f1-d0914d6eadc2") : object "kube-system"/"original-pull-secret" not registered
Jul 14 00:59:48.807974 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:48.807925    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5nj\" (UniqueName: \"kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj\") pod \"network-check-target-6dnxf\" (UID: \"455ca3e1-0a9d-4b8e-b150-06ba86301ae5\") " pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:48.808127 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:48.808089    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 00:59:48.808127 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:48.808109    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 00:59:48.808127 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:48.808119    2567 projected.go:196] Error preparing data for projected volume kube-api-access-gg5nj for pod openshift-network-diagnostics/network-check-target-6dnxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:59:48.808260 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:48.808179    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj podName:455ca3e1-0a9d-4b8e-b150-06ba86301ae5 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:04.808161471 +0000 UTC m=+34.372570509 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gg5nj" (UniqueName: "kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj") pod "network-check-target-6dnxf" (UID: "455ca3e1-0a9d-4b8e-b150-06ba86301ae5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:59:48.969078 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:48.969039    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:48.969253 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:48.969182    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 00:59:49.968747 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:49.968709    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:49.969114 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:49.968710    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:49.969114 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:49.968829    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 00:59:49.969114 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:49.968909    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 00:59:50.969594 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:50.969554    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:50.970053 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:50.969669    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 00:59:51.969354 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:51.969313    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:51.969543 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:51.969367    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:51.969543 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:51.969467    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 00:59:51.969647 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:51.969591    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 00:59:52.969267 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:52.969232    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:52.969494 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:52.969343    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 00:59:53.968731 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:53.968502    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:53.969211 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:53.968842    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 00:59:53.969211 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:53.968668    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:53.969211 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:53.969026    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 00:59:54.169056 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.168855    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jpnj7" event={"ID":"b5459bba-8c2e-4be1-802f-a462a23327f6","Type":"ContainerStarted","Data":"9a3372d8e776319fd80e4515dd1ff840edf5af1505540f4380dd395e6b5ca0d3"}
Jul 14 00:59:54.171301 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.171266    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd" event={"ID":"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45","Type":"ContainerStarted","Data":"b244b1cab1f77479325c90315b5994c2a433d4c56ef7e62d1155be3c4a697531"}
Jul 14 00:59:54.171470 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.171360    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd" event={"ID":"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45","Type":"ContainerStarted","Data":"5b643a2fdd0e5938f888df7858c566d6327cc475b12b422221fbbb6923560131"}
Jul 14 00:59:54.172853 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.172813    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ggmtc" event={"ID":"a58647a6-7a3b-4298-b6b5-a18848141893","Type":"ContainerStarted","Data":"15cef630f78ddf9fb8c9cca696bb20e5e809144e413113730c67743c0eb0841d"}
Jul 14 00:59:54.174345 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.174311    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d4fkp" event={"ID":"2998116f-cba6-4d84-a68c-c4da115286ad","Type":"ContainerStarted","Data":"fd2bdfba5b0de159359954102f177fd48084d20408eb06aa03d7ba7992604317"}
Jul 14 00:59:54.175850 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.175826    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6" event={"ID":"bfc53be7-89a1-4e54-9188-a194c7a4c095","Type":"ContainerStarted","Data":"da5a8a04361af9b809a5f58f7789768dee20358db73130d1cbee4282bb278c7a"}
Jul 14 00:59:54.177122 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.177101    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2z2tl" event={"ID":"336250ce-5e32-48f2-88a0-365516e9b4b7","Type":"ContainerStarted","Data":"d87e9dd40d286ce13275df8bdbdf3d7e78cf51830176383fe6d9ad01a3aa13fe"}
Jul 14 00:59:54.180413 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.180389    2567 generic.go:356] "Generic (PLEG): container finished" podID="31b7f78f-0db2-469f-886d-a83f025c6a62" containerID="38a0929a807f48f0e0b8463cadf5f65bec327023aeef33f16480b7080d00088d" exitCode=0
Jul 14 00:59:54.180519 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.180455    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cg7w8" event={"ID":"31b7f78f-0db2-469f-886d-a83f025c6a62","Type":"ContainerDied","Data":"38a0929a807f48f0e0b8463cadf5f65bec327023aeef33f16480b7080d00088d"}
Jul 14 00:59:54.182375 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.182103    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2mx5v" event={"ID":"3a144e7e-6d83-49a5-90c8-6052b6734d80","Type":"ContainerStarted","Data":"47530ede401924b009bc75e01086f487b64c435b020708ec58a61d633d8de488"}
Jul 14 00:59:54.183308 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.183224    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jpnj7" podStartSLOduration=3.612360644 podStartE2EDuration="23.183213148s" totalImagesPullingTime="19.570852504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:59:54.183078738 +0000 UTC m=+23.747487793" watchObservedRunningTime="2026-07-14 00:59:54.183213148 +0000 UTC m=+23.747622203"
Jul 14 00:59:54.200975 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.199147    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ggmtc" podStartSLOduration=3.57064665 podStartE2EDuration="23.199123159s" totalImagesPullingTime="19.628476509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:59:54.197744917 +0000 UTC m=+23.762153973" watchObservedRunningTime="2026-07-14 00:59:54.199123159 +0000 UTC m=+23.763532215"
Jul 14 00:59:54.247073 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.247029    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d4fkp" podStartSLOduration=2.665569814 podStartE2EDuration="22.247011711s" totalImagesPullingTime="19.581441897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:59:54.246754224 +0000 UTC m=+23.811163281" watchObservedRunningTime="2026-07-14 00:59:54.247011711 +0000 UTC m=+23.811420767"
Jul 14 00:59:54.262165 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.262116    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2mx5v" podStartSLOduration=3.603161833 podStartE2EDuration="23.262099746s" totalImagesPullingTime="19.658937913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:59:54.261593482 +0000 UTC m=+23.826002536" watchObservedRunningTime="2026-07-14 00:59:54.262099746 +0000 UTC m=+23.826508802"
Jul 14 00:59:54.274604 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.274550    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2z2tl" podStartSLOduration=7.420409108 podStartE2EDuration="23.274529106s" totalImagesPullingTime="15.854119998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:59:54.273981607 +0000 UTC m=+23.838390652" watchObservedRunningTime="2026-07-14 00:59:54.274529106 +0000 UTC m=+23.838938160"
Jul 14 00:59:54.969179 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:54.969139    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:54.969908 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:54.969265    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 00:59:55.127860 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.127810    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2z2tl"
Jul 14 00:59:55.128462 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.128443    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2z2tl"
Jul 14 00:59:55.188774 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.188744    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd" event={"ID":"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45","Type":"ContainerStarted","Data":"40c798d7681b85684da06552ee72786ca7bec40e66f0ed644462790ae0566e99"}
Jul 14 00:59:55.188966 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.188824    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd" event={"ID":"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45","Type":"ContainerStarted","Data":"ddcf721b6fe49acf70e508dfe9a00d87e32b0ed6ba71ee6a9fc1a07dc94ab921"}
Jul 14 00:59:55.188966 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.188877    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd" event={"ID":"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45","Type":"ContainerStarted","Data":"6c0083ad77680f2ad43e977d46d7d0f5fac9f0544acf2f3f5e3eb1d924019776"}
Jul 14 00:59:55.188966 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.188927    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd" event={"ID":"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45","Type":"ContainerStarted","Data":"d1bb1850b1235985323e761977df7b676f696bdb1535f7e0eb0781439fafc4d0"}
Jul 14 00:59:55.191027 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.190503    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hlqcw" event={"ID":"0d726e56-7cd0-49b6-9d1e-3949bdc47970","Type":"ContainerStarted","Data":"884da7b24e83f611cabddb28d722cf923f88b320d664b83ae7902764ae2cbe8c"}
Jul 14 00:59:55.191600 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.191223    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2z2tl"
Jul 14 00:59:55.191829 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.191810    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2z2tl"
Jul 14 00:59:55.204443 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.204379    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hlqcw" podStartSLOduration=3.628087458 podStartE2EDuration="23.204355181s" totalImagesPullingTime="19.576267723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:59:55.20375171 +0000 UTC m=+24.768160766" watchObservedRunningTime="2026-07-14 00:59:55.204355181 +0000 UTC m=+24.768764237"
Jul 14 00:59:55.286723 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.286684    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 14 00:59:55.925123 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.925015    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T00:59:55.286707008Z","UUID":"c5f408cf-39b8-47af-8991-a8987664008a","Handler":null,"Name":"","Endpoint":""}
Jul 14 00:59:55.928216 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.928192    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 14 00:59:55.928364 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.928236    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 14 00:59:55.969190 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.968640    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:55.969190 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:55.968793    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 00:59:55.969190 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:55.968849    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:55.969190 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:55.968978    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 00:59:56.195005 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:56.194897    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6" event={"ID":"bfc53be7-89a1-4e54-9188-a194c7a4c095","Type":"ContainerStarted","Data":"d6001ed8e85e105dc0401291d96352804735307a71b99ba3d09aa9cc5332f137"}
Jul 14 00:59:56.968851 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:56.968573    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:56.969103 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:56.968917    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 00:59:57.201412 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:57.201365    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd" event={"ID":"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45","Type":"ContainerStarted","Data":"6d056ea9bdb5fd3ca52ec1e97273c922ccef6ea6cf9729637b048319ca9ba4e0"}
Jul 14 00:59:57.203725 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:57.203697    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6" event={"ID":"bfc53be7-89a1-4e54-9188-a194c7a4c095","Type":"ContainerStarted","Data":"5da6c73f10d6c885ef941e131432062a5317e8c56104e79ed6d4434287feb591"}
Jul 14 00:59:57.222451 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:57.222351    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2v4f6" podStartSLOduration=8.230965664 podStartE2EDuration="26.222331679s" totalImagesPullingTime="17.991366015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:59:57.2213756 +0000 UTC m=+26.785784656" watchObservedRunningTime="2026-07-14 00:59:57.222331679 +0000 UTC m=+26.786740736"
Jul 14 00:59:57.968974 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:57.968931    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:57.969185 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:57.968931    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:57.969185 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:57.969058    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 00:59:57.969185 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:57.969147    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 00:59:58.969289 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:58.969254    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 00:59:58.969910 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:58.969434    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 00:59:59.212695 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:59.212374    2567 generic.go:356] "Generic (PLEG): container finished" podID="31b7f78f-0db2-469f-886d-a83f025c6a62" containerID="a7c0c894b7b6d5575183e7cd179fdab741dc598adda4afa41d3497472ac3a1c8" exitCode=0
Jul 14 00:59:59.212695 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:59.212458    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cg7w8" event={"ID":"31b7f78f-0db2-469f-886d-a83f025c6a62","Type":"ContainerDied","Data":"a7c0c894b7b6d5575183e7cd179fdab741dc598adda4afa41d3497472ac3a1c8"}
Jul 14 00:59:59.216420 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:59.216383    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd" event={"ID":"ca6d2ac7-b806-485c-b968-f1e2bb8a1a45","Type":"ContainerStarted","Data":"1adc97679438e677b7dc96c527948a119015c69dd42e38b5a3d36fd462ffb195"}
Jul 14 00:59:59.216683 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:59.216661    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:59.234017 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:59.233988    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 00:59:59.253678 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:59.253624    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd" podStartSLOduration=8.59266305 podStartE2EDuration="28.253608819s" totalImagesPullingTime="19.660945769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:59:59.253298723 +0000 UTC m=+28.817707777" watchObservedRunningTime="2026-07-14 00:59:59.253608819 +0000 UTC m=+28.818017874"
Jul 14 00:59:59.969375 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:59.969340    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 00:59:59.969801 ip-10-0-129-117 kubenswrapper[2567]: I0714 00:59:59.969399    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 00:59:59.969801 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:59.969498    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 00:59:59.969801 ip-10-0-129-117 kubenswrapper[2567]: E0714 00:59:59.969623    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 01:00:00.219083 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:00.219051    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 01:00:00.219257 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:00.219122    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 01:00:00.245340 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:00.239891    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 01:00:00.952319 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:00.952288    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q8j4f"]
Jul 14 01:00:00.952551 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:00.952398    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 01:00:00.952551 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:00.952480    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 01:00:00.955377 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:00.955144    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6dnxf"]
Jul 14 01:00:00.955377 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:00.955281    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 01:00:00.955562 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:00.955378    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 01:00:00.955818 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:00.955797    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jfg2f"]
Jul 14 01:00:00.955903 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:00.955892    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 01:00:00.956027 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:00.956006    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 01:00:01.222600 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:01.222508    2567 generic.go:356] "Generic (PLEG): container finished" podID="31b7f78f-0db2-469f-886d-a83f025c6a62" containerID="b780a727fa7cff441e542295076d562bf5d5455fff8f1848f697b1f4acf06c9d" exitCode=0
Jul 14 01:00:01.222600 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:01.222566    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cg7w8" event={"ID":"31b7f78f-0db2-469f-886d-a83f025c6a62","Type":"ContainerDied","Data":"b780a727fa7cff441e542295076d562bf5d5455fff8f1848f697b1f4acf06c9d"}
Jul 14 01:00:02.969094 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:02.969056    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 01:00:02.969552 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:02.969066    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 01:00:02.969552 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:02.969197    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 01:00:02.969552 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:02.969068    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 01:00:02.969552 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:02.969257    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 01:00:02.969552 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:02.969323    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 01:00:04.230709 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:04.230671    2567 generic.go:356] "Generic (PLEG): container finished" podID="31b7f78f-0db2-469f-886d-a83f025c6a62" containerID="79b9683ef4ae0513583363c0786cc074e28786bf63cb39907d2b2ce278992702" exitCode=0
Jul 14 01:00:04.231296 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:04.230726    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cg7w8" event={"ID":"31b7f78f-0db2-469f-886d-a83f025c6a62","Type":"ContainerDied","Data":"79b9683ef4ae0513583363c0786cc074e28786bf63cb39907d2b2ce278992702"}
Jul 14 01:00:04.627757 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:04.627716    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 01:00:04.627908 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:04.627863    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:04.627994 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:04.627928    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs podName:f0ebac4e-8f5b-4820-b018-8386d03067f1 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:36.62791337 +0000 UTC m=+66.192322403 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs") pod "network-metrics-daemon-jfg2f" (UID: "f0ebac4e-8f5b-4820-b018-8386d03067f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:04.728238 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:04.728184    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 01:00:04.728404 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:04.728337    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:04.728404 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:04.728400    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret podName:1a747595-65ad-41fe-a1f1-d0914d6eadc2 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:36.728384749 +0000 UTC m=+66.292793782 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret") pod "global-pull-secret-syncer-q8j4f" (UID: "1a747595-65ad-41fe-a1f1-d0914d6eadc2") : object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:04.829400 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:04.829361    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5nj\" (UniqueName: \"kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj\") pod \"network-check-target-6dnxf\" (UID: \"455ca3e1-0a9d-4b8e-b150-06ba86301ae5\") " pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 01:00:04.829563 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:04.829526    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 01:00:04.829563 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:04.829548    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 01:00:04.829563 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:04.829559    2567 projected.go:196] Error preparing data for projected volume kube-api-access-gg5nj for pod openshift-network-diagnostics/network-check-target-6dnxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:04.829676 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:04.829610    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj podName:455ca3e1-0a9d-4b8e-b150-06ba86301ae5 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:36.82959682 +0000 UTC m=+66.394005854 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gg5nj" (UniqueName: "kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj") pod "network-check-target-6dnxf" (UID: "455ca3e1-0a9d-4b8e-b150-06ba86301ae5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:04.969513 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:04.969431    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 01:00:04.969655 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:04.969431    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 01:00:04.969655 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:04.969557    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 01:00:04.969655 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:04.969640    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8j4f" podUID="1a747595-65ad-41fe-a1f1-d0914d6eadc2"
Jul 14 01:00:04.969763 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:04.969431    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 01:00:04.969763 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:04.969739    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnxf" podUID="455ca3e1-0a9d-4b8e-b150-06ba86301ae5"
Jul 14 01:00:06.706313 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.706275    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeReady"
Jul 14 01:00:06.706858 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.706445    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 14 01:00:06.740732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.740695    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d59cbd55f-nfdch"]
Jul 14 01:00:06.742919 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.742888    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"]
Jul 14 01:00:06.743067 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.742932    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.744781 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.744758    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:00:06.744781 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.744772    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 14 01:00:06.744978 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.744919    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 14 01:00:06.745313 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.745288    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 14 01:00:06.745418 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.745397    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5xx4b\""
Jul 14 01:00:06.746717 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.746688    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 14 01:00:06.747051 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.747031    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 14 01:00:06.747051 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.747046    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-sp2ms\""
Jul 14 01:00:06.751093 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.751070    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 14 01:00:06.754941 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.754916    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"]
Jul 14 01:00:06.760832 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.760799    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ghmc7"]
Jul 14 01:00:06.763098 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.763072    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d59cbd55f-nfdch"]
Jul 14 01:00:06.763295 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.763280    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jjkx4"]
Jul 14 01:00:06.764042 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.763205    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ghmc7"
Jul 14 01:00:06.765463 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.765439    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:06.766364 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.766328    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 14 01:00:06.766364 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.766333    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 14 01:00:06.766364 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.766362    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 14 01:00:06.766583 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.766393    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qwwjg\""
Jul 14 01:00:06.767373 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.767354    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 14 01:00:06.767453 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.767415    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 14 01:00:06.767539 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.767526    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jf2dk\""
Jul 14 01:00:06.774317 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.774217    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ghmc7"]
Jul 14 01:00:06.774317 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.774278    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jjkx4"]
Jul 14 01:00:06.844428 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.844385    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f83ab17-7261-408c-aaad-f38a36de4213-image-registry-private-configuration\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.844599 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.844444    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f83ab17-7261-408c-aaad-f38a36de4213-ca-trust-extracted\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.844599 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.844579    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-bound-sa-token\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.844712 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.844632    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdc99\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-kube-api-access-pdc99\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.844712 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.844702    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f83ab17-7261-408c-aaad-f38a36de4213-registry-certificates\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.844794 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.844735    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jhd7p\" (UID: \"66ca18b4-5345-4c49-9772-e4bfd1bc6747\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:00:06.844794 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.844777    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.844876 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.844843    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f83ab17-7261-408c-aaad-f38a36de4213-installation-pull-secrets\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.844914 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.844880    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f83ab17-7261-408c-aaad-f38a36de4213-trusted-ca\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.844914 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.844906    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/66ca18b4-5345-4c49-9772-e4bfd1bc6747-nginx-conf\") pod \"networking-console-plugin-6cf856758c-jhd7p\" (UID: \"66ca18b4-5345-4c49-9772-e4bfd1bc6747\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:00:06.946148 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946106    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f83ab17-7261-408c-aaad-f38a36de4213-ca-trust-extracted\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.946323 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946184    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-bound-sa-token\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.946323 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946213    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdc99\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-kube-api-access-pdc99\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.946323 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946252    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f83ab17-7261-408c-aaad-f38a36de4213-registry-certificates\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.946323 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946281    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8f406c4-b03c-48ff-a9d0-d12af1c70dff-cert\") pod \"ingress-canary-ghmc7\" (UID: \"c8f406c4-b03c-48ff-a9d0-d12af1c70dff\") " pod="openshift-ingress-canary/ingress-canary-ghmc7"
Jul 14 01:00:06.946323 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946313    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jhd7p\" (UID: \"66ca18b4-5345-4c49-9772-e4bfd1bc6747\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:00:06.946561 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946337    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-tmp-dir\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:06.946561 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946372    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.946561 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946400    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f83ab17-7261-408c-aaad-f38a36de4213-installation-pull-secrets\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.946561 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946427    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f83ab17-7261-408c-aaad-f38a36de4213-trusted-ca\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.946561 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946450    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-config-volume\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:06.946561 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946475    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g5qg\" (UniqueName: \"kubernetes.io/projected/c8f406c4-b03c-48ff-a9d0-d12af1c70dff-kube-api-access-4g5qg\") pod \"ingress-canary-ghmc7\" (UID: \"c8f406c4-b03c-48ff-a9d0-d12af1c70dff\") " pod="openshift-ingress-canary/ingress-canary-ghmc7"
Jul 14 01:00:06.946561 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946495    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:06.946561 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946513    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/66ca18b4-5345-4c49-9772-e4bfd1bc6747-nginx-conf\") pod \"networking-console-plugin-6cf856758c-jhd7p\" (UID: \"66ca18b4-5345-4c49-9772-e4bfd1bc6747\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:00:06.946561 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946541    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k2gv\" (UniqueName: \"kubernetes.io/projected/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-kube-api-access-8k2gv\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:06.946561 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946549    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f83ab17-7261-408c-aaad-f38a36de4213-ca-trust-extracted\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.946949 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:06.946624    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 01:00:06.946949 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:06.946635    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d59cbd55f-nfdch: secret "image-registry-tls" not found
Jul 14 01:00:06.946949 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:06.946690    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls podName:4f83ab17-7261-408c-aaad-f38a36de4213 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:07.446676789 +0000 UTC m=+37.011085822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls") pod "image-registry-6d59cbd55f-nfdch" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213") : secret "image-registry-tls" not found
Jul 14 01:00:06.946949 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946920    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f83ab17-7261-408c-aaad-f38a36de4213-registry-certificates\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.947100 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:06.947028    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 14 01:00:06.947100 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:06.947068    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert podName:66ca18b4-5345-4c49-9772-e4bfd1bc6747 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:07.447056409 +0000 UTC m=+37.011465442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-jhd7p" (UID: "66ca18b4-5345-4c49-9772-e4bfd1bc6747") : secret "networking-console-plugin-cert" not found
Jul 14 01:00:06.947100 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.946560    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f83ab17-7261-408c-aaad-f38a36de4213-image-registry-private-configuration\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.947510 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.947471    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/66ca18b4-5345-4c49-9772-e4bfd1bc6747-nginx-conf\") pod \"networking-console-plugin-6cf856758c-jhd7p\" (UID: \"66ca18b4-5345-4c49-9772-e4bfd1bc6747\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:00:06.947919 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.947896    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f83ab17-7261-408c-aaad-f38a36de4213-trusted-ca\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.951388 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.951360    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f83ab17-7261-408c-aaad-f38a36de4213-image-registry-private-configuration\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.951388 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.951370    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f83ab17-7261-408c-aaad-f38a36de4213-installation-pull-secrets\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.954154 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.954133    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-bound-sa-token\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.954464 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.954445    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdc99\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-kube-api-access-pdc99\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:06.969124 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.969053    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 01:00:06.969261 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.969053    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 01:00:06.969261 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.969067    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 01:00:06.971173 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.971149    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 01:00:06.971173 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.971164    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 01:00:06.971907 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.971643    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 01:00:06.971907 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.971751    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 01:00:06.971907 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.971790    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z2php\""
Jul 14 01:00:06.971907 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:06.971897    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6f2s8\""
Jul 14 01:00:07.048500 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.048460    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8f406c4-b03c-48ff-a9d0-d12af1c70dff-cert\") pod \"ingress-canary-ghmc7\" (UID: \"c8f406c4-b03c-48ff-a9d0-d12af1c70dff\") " pod="openshift-ingress-canary/ingress-canary-ghmc7"
Jul 14 01:00:07.048692 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.048528    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-tmp-dir\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:07.048692 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.048583    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-config-volume\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:07.048692 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.048608    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g5qg\" (UniqueName: \"kubernetes.io/projected/c8f406c4-b03c-48ff-a9d0-d12af1c70dff-kube-api-access-4g5qg\") pod \"ingress-canary-ghmc7\" (UID: \"c8f406c4-b03c-48ff-a9d0-d12af1c70dff\") " pod="openshift-ingress-canary/ingress-canary-ghmc7"
Jul 14 01:00:07.048692 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.048631    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:07.048692 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.048675    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k2gv\" (UniqueName: \"kubernetes.io/projected/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-kube-api-access-8k2gv\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:07.048977 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.048932    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-tmp-dir\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:07.049056 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:07.049041    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 01:00:07.049123 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:07.049113    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls podName:ef8fc61b-7241-4cec-bda8-ecfb2ed789f1 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:07.549096894 +0000 UTC m=+37.113505932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls") pod "dns-default-jjkx4" (UID: "ef8fc61b-7241-4cec-bda8-ecfb2ed789f1") : secret "dns-default-metrics-tls" not found
Jul 14 01:00:07.049220 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.049195    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-config-volume\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:07.051816 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.051789    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8f406c4-b03c-48ff-a9d0-d12af1c70dff-cert\") pod \"ingress-canary-ghmc7\" (UID: \"c8f406c4-b03c-48ff-a9d0-d12af1c70dff\") " pod="openshift-ingress-canary/ingress-canary-ghmc7"
Jul 14 01:00:07.058680 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.058644    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k2gv\" (UniqueName: \"kubernetes.io/projected/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-kube-api-access-8k2gv\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:07.059127 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.059099    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g5qg\" (UniqueName: \"kubernetes.io/projected/c8f406c4-b03c-48ff-a9d0-d12af1c70dff-kube-api-access-4g5qg\") pod \"ingress-canary-ghmc7\" (UID: \"c8f406c4-b03c-48ff-a9d0-d12af1c70dff\") " pod="openshift-ingress-canary/ingress-canary-ghmc7"
Jul 14 01:00:07.078065 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.078031    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ghmc7"
Jul 14 01:00:07.265175 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.264841    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ghmc7"]
Jul 14 01:00:07.270602 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:00:07.270576    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f406c4_b03c_48ff_a9d0_d12af1c70dff.slice/crio-2110cfb1b2b8788a314dc6b907e70dfd73d2790f4962775ad6642c5f10074a34 WatchSource:0}: Error finding container 2110cfb1b2b8788a314dc6b907e70dfd73d2790f4962775ad6642c5f10074a34: Status 404 returned error can't find the container with id 2110cfb1b2b8788a314dc6b907e70dfd73d2790f4962775ad6642c5f10074a34
Jul 14 01:00:07.285654 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.285613    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:07.376116 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.376078    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:07.445794 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.445756    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:07.452765 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.452736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jhd7p\" (UID: \"66ca18b4-5345-4c49-9772-e4bfd1bc6747\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:00:07.452966 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.452779    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:07.452966 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:07.452886    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 14 01:00:07.452966 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:07.452908    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 01:00:07.452966 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:07.452919    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d59cbd55f-nfdch: secret "image-registry-tls" not found
Jul 14 01:00:07.453187 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:07.452970    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert podName:66ca18b4-5345-4c49-9772-e4bfd1bc6747 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:08.452933236 +0000 UTC m=+38.017342270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-jhd7p" (UID: "66ca18b4-5345-4c49-9772-e4bfd1bc6747") : secret "networking-console-plugin-cert" not found
Jul 14 01:00:07.453187 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:07.452989    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls podName:4f83ab17-7261-408c-aaad-f38a36de4213 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:08.452979608 +0000 UTC m=+38.017388648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls") pod "image-registry-6d59cbd55f-nfdch" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213") : secret "image-registry-tls" not found
Jul 14 01:00:07.554283 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.554199    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:07.554443 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:07.554322    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 01:00:07.554443 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:07.554385    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls podName:ef8fc61b-7241-4cec-bda8-ecfb2ed789f1 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:08.554369093 +0000 UTC m=+38.118778126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls") pod "dns-default-jjkx4" (UID: "ef8fc61b-7241-4cec-bda8-ecfb2ed789f1") : secret "dns-default-metrics-tls" not found
Jul 14 01:00:07.560387 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.560341    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:07.661908 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:07.661870    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:08.242925 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:08.242886    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ghmc7" event={"ID":"c8f406c4-b03c-48ff-a9d0-d12af1c70dff","Type":"ContainerStarted","Data":"2110cfb1b2b8788a314dc6b907e70dfd73d2790f4962775ad6642c5f10074a34"}
Jul 14 01:00:08.290557 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:08.290521    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:08.462141 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:08.462103    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jhd7p\" (UID: \"66ca18b4-5345-4c49-9772-e4bfd1bc6747\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:00:08.462335 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:08.462163    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:08.462335 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:08.462239    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 14 01:00:08.462335 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:08.462291    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 01:00:08.462335 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:08.462310    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d59cbd55f-nfdch: secret "image-registry-tls" not found
Jul 14 01:00:08.462335 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:08.462322    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert podName:66ca18b4-5345-4c49-9772-e4bfd1bc6747 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:10.4623022 +0000 UTC m=+40.026711237 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-jhd7p" (UID: "66ca18b4-5345-4c49-9772-e4bfd1bc6747") : secret "networking-console-plugin-cert" not found
Jul 14 01:00:08.462588 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:08.462355    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls podName:4f83ab17-7261-408c-aaad-f38a36de4213 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:10.462343189 +0000 UTC m=+40.026752226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls") pod "image-registry-6d59cbd55f-nfdch" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213") : secret "image-registry-tls" not found
Jul 14 01:00:08.563112 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:08.563008    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:08.563265 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:08.563139    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 01:00:08.563265 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:08.563206    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls podName:ef8fc61b-7241-4cec-bda8-ecfb2ed789f1 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:10.563189593 +0000 UTC m=+40.127598628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls") pod "dns-default-jjkx4" (UID: "ef8fc61b-7241-4cec-bda8-ecfb2ed789f1") : secret "dns-default-metrics-tls" not found
Jul 14 01:00:08.753659 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:08.753617    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:08.808628 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:08.808573    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:09.794721 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:09.794682    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:09.873456 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:09.873384    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:09.975401 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:09.975361    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:10.254210 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:10.254175    2567 generic.go:356] "Generic (PLEG): container finished" podID="31b7f78f-0db2-469f-886d-a83f025c6a62" containerID="b54c27a068634ac5994937e555e8b57246fed14accff6330bedda7c543d65976" exitCode=0
Jul 14 01:00:10.254351 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:10.254237    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cg7w8" event={"ID":"31b7f78f-0db2-469f-886d-a83f025c6a62","Type":"ContainerDied","Data":"b54c27a068634ac5994937e555e8b57246fed14accff6330bedda7c543d65976"}
Jul 14 01:00:10.482388 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:10.482344    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jhd7p\" (UID: \"66ca18b4-5345-4c49-9772-e4bfd1bc6747\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:00:10.482564 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:10.482399    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:10.482564 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:10.482499    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 14 01:00:10.482564 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:10.482526    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 01:00:10.482564 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:10.482544    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d59cbd55f-nfdch: secret "image-registry-tls" not found
Jul 14 01:00:10.482733 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:10.482575    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert podName:66ca18b4-5345-4c49-9772-e4bfd1bc6747 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:14.48255804 +0000 UTC m=+44.046967074 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-jhd7p" (UID: "66ca18b4-5345-4c49-9772-e4bfd1bc6747") : secret "networking-console-plugin-cert" not found
Jul 14 01:00:10.482733 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:10.482595    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls podName:4f83ab17-7261-408c-aaad-f38a36de4213 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:14.48258253 +0000 UTC m=+44.046991562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls") pod "image-registry-6d59cbd55f-nfdch" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213") : secret "image-registry-tls" not found
Jul 14 01:00:10.583685 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:10.583643    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:10.583845 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:10.583812    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 01:00:10.583908 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:10.583896    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls podName:ef8fc61b-7241-4cec-bda8-ecfb2ed789f1 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:14.583871611 +0000 UTC m=+44.148280652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls") pod "dns-default-jjkx4" (UID: "ef8fc61b-7241-4cec-bda8-ecfb2ed789f1") : secret "dns-default-metrics-tls" not found
Jul 14 01:00:10.684432 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:10.684378    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:10.732317 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:10.732280    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:11.091456 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:11.091428    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:11.143002 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:11.142933    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:11.259576 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:11.259537    2567 generic.go:356] "Generic (PLEG): container finished" podID="31b7f78f-0db2-469f-886d-a83f025c6a62" containerID="8e35164f329abb09b53af8db232e07c64f345db7d68b3159b6d69948b8f994cf" exitCode=0
Jul 14 01:00:11.259744 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:11.259611    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cg7w8" event={"ID":"31b7f78f-0db2-469f-886d-a83f025c6a62","Type":"ContainerDied","Data":"8e35164f329abb09b53af8db232e07c64f345db7d68b3159b6d69948b8f994cf"}
Jul 14 01:00:12.265020 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:12.264983    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cg7w8" event={"ID":"31b7f78f-0db2-469f-886d-a83f025c6a62","Type":"ContainerStarted","Data":"5a8b6d6942ac053a69ce2bb264cefbd4a7658341cce0991f82d7c54992047f52"}
Jul 14 01:00:12.266479 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:12.266453    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ghmc7" event={"ID":"c8f406c4-b03c-48ff-a9d0-d12af1c70dff","Type":"ContainerStarted","Data":"ad73cef4066916f6e879c2c2ccfe25514bb2d9e757a857c73e2cd4d17d8b2d9f"}
Jul 14 01:00:12.286622 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:12.286572    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cg7w8" podStartSLOduration=8.798455407 podStartE2EDuration="41.286556786s" totalImagesPullingTime="32.488101379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:12.285572395 +0000 UTC m=+41.849981450" watchObservedRunningTime="2026-07-14 01:00:12.286556786 +0000 UTC m=+41.850965838"
Jul 14 01:00:12.303206 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:12.303145    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ghmc7" podStartSLOduration=2.427728216 podStartE2EDuration="6.303128135s" totalImagesPullingTime="3.875399919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:12.301877905 +0000 UTC m=+41.866286961" watchObservedRunningTime="2026-07-14 01:00:12.303128135 +0000 UTC m=+41.867537245"
Jul 14 01:00:13.007230 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:13.007196    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:13.434343 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:13.434309    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:13.626947 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:13.626910    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:13.728249 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:13.728170    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:14.516562 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:14.516517    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jhd7p\" (UID: \"66ca18b4-5345-4c49-9772-e4bfd1bc6747\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:00:14.516562 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:14.516564    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:14.517025 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:14.516661    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 01:00:14.517025 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:14.516672    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d59cbd55f-nfdch: secret "image-registry-tls" not found
Jul 14 01:00:14.517025 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:14.516677    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 14 01:00:14.517025 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:14.516723    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls podName:4f83ab17-7261-408c-aaad-f38a36de4213 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:22.516708672 +0000 UTC m=+52.081117704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls") pod "image-registry-6d59cbd55f-nfdch" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213") : secret "image-registry-tls" not found
Jul 14 01:00:14.517025 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:14.516745    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert podName:66ca18b4-5345-4c49-9772-e4bfd1bc6747 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:22.516730722 +0000 UTC m=+52.081139756 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-jhd7p" (UID: "66ca18b4-5345-4c49-9772-e4bfd1bc6747") : secret "networking-console-plugin-cert" not found
Jul 14 01:00:14.617294 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:14.617259    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:14.617428 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:14.617410    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 01:00:14.617488 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:14.617478    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls podName:ef8fc61b-7241-4cec-bda8-ecfb2ed789f1 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:22.617460714 +0000 UTC m=+52.181869749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls") pod "dns-default-jjkx4" (UID: "ef8fc61b-7241-4cec-bda8-ecfb2ed789f1") : secret "dns-default-metrics-tls" not found
Jul 14 01:00:22.581185 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:22.581146    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jhd7p\" (UID: \"66ca18b4-5345-4c49-9772-e4bfd1bc6747\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:00:22.581185 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:22.581193    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:22.581757 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:22.581303    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 01:00:22.581757 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:22.581304    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 14 01:00:22.581757 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:22.581374    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert podName:66ca18b4-5345-4c49-9772-e4bfd1bc6747 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:38.581356514 +0000 UTC m=+68.145765547 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-jhd7p" (UID: "66ca18b4-5345-4c49-9772-e4bfd1bc6747") : secret "networking-console-plugin-cert" not found
Jul 14 01:00:22.581757 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:22.581314    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d59cbd55f-nfdch: secret "image-registry-tls" not found
Jul 14 01:00:22.581757 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:22.581406    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls podName:4f83ab17-7261-408c-aaad-f38a36de4213 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:38.581399357 +0000 UTC m=+68.145808390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls") pod "image-registry-6d59cbd55f-nfdch" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213") : secret "image-registry-tls" not found
Jul 14 01:00:22.681591 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:22.681554    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:22.681744 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:22.681670    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 01:00:22.681744 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:22.681729    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls podName:ef8fc61b-7241-4cec-bda8-ecfb2ed789f1 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:38.681715834 +0000 UTC m=+68.246124867 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls") pod "dns-default-jjkx4" (UID: "ef8fc61b-7241-4cec-bda8-ecfb2ed789f1") : secret "dns-default-metrics-tls" not found
Jul 14 01:00:29.386332 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:29.386286    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:29.439617 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:29.439578    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:29.491996 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:29.491934    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:30.401455 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:30.401417    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:30.501534 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:30.501503    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:32.236220 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:32.236183    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6kbd"
Jul 14 01:00:32.383193 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:32.383154    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:32.482262 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:32.482219    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:32.808429 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:32.808392    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:32.869266 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:32.869224    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:33.705888 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:33.705849    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:33.809058 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:33.809019    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:33.908474 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:33.908438    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:34.009160 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:34.009077    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:36.688309 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:36.688253    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 01:00:36.690273 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:36.690252    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 01:00:36.698833 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:36.698807    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 14 01:00:36.698914 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:36.698884    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs podName:f0ebac4e-8f5b-4820-b018-8386d03067f1 nodeName:}" failed. No retries permitted until 2026-07-14 01:01:40.698861549 +0000 UTC m=+130.263270583 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs") pod "network-metrics-daemon-jfg2f" (UID: "f0ebac4e-8f5b-4820-b018-8386d03067f1") : secret "metrics-daemon-secret" not found
Jul 14 01:00:36.789278 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:36.789236    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 01:00:36.791281 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:36.791258    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 01:00:36.803701 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:36.803668    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a747595-65ad-41fe-a1f1-d0914d6eadc2-original-pull-secret\") pod \"global-pull-secret-syncer-q8j4f\" (UID: \"1a747595-65ad-41fe-a1f1-d0914d6eadc2\") " pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 01:00:36.809287 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:36.809258    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:36.890168 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:36.890132    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5nj\" (UniqueName: \"kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj\") pod \"network-check-target-6dnxf\" (UID: \"455ca3e1-0a9d-4b8e-b150-06ba86301ae5\") " pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 01:00:36.892529 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:36.892505    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 01:00:36.902601 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:36.902575    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 01:00:36.914791 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:36.914760    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5nj\" (UniqueName: \"kubernetes.io/projected/455ca3e1-0a9d-4b8e-b150-06ba86301ae5-kube-api-access-gg5nj\") pod \"network-check-target-6dnxf\" (UID: \"455ca3e1-0a9d-4b8e-b150-06ba86301ae5\") " pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 01:00:36.981634 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:36.981547    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8j4f"
Jul 14 01:00:36.989720 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:36.989685    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z2php\""
Jul 14 01:00:36.998368 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:36.998336    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 01:00:37.121215 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:37.121143    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q8j4f"]
Jul 14 01:00:37.125086 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:00:37.125056    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a747595_65ad_41fe_a1f1_d0914d6eadc2.slice/crio-943054781b519f54d2a1b4eecb09ac8e4456fac5a59f3b0f2c5edfcf5dbafa9d WatchSource:0}: Error finding container 943054781b519f54d2a1b4eecb09ac8e4456fac5a59f3b0f2c5edfcf5dbafa9d: Status 404 returned error can't find the container with id 943054781b519f54d2a1b4eecb09ac8e4456fac5a59f3b0f2c5edfcf5dbafa9d
Jul 14 01:00:37.144297 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:37.144263    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6dnxf"]
Jul 14 01:00:37.147987 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:00:37.147938    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455ca3e1_0a9d_4b8e_b150_06ba86301ae5.slice/crio-00083925d10320c7eddc22800691a8aa45127b0a323ca3d926988d082a44d8c0 WatchSource:0}: Error finding container 00083925d10320c7eddc22800691a8aa45127b0a323ca3d926988d082a44d8c0: Status 404 returned error can't find the container with id 00083925d10320c7eddc22800691a8aa45127b0a323ca3d926988d082a44d8c0
Jul 14 01:00:37.326206 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:37.326111    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6dnxf" event={"ID":"455ca3e1-0a9d-4b8e-b150-06ba86301ae5","Type":"ContainerStarted","Data":"00083925d10320c7eddc22800691a8aa45127b0a323ca3d926988d082a44d8c0"}
Jul 14 01:00:37.327060 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:37.327033    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q8j4f" event={"ID":"1a747595-65ad-41fe-a1f1-d0914d6eadc2","Type":"ContainerStarted","Data":"943054781b519f54d2a1b4eecb09ac8e4456fac5a59f3b0f2c5edfcf5dbafa9d"}
Jul 14 01:00:38.604012 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:38.603942    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jhd7p\" (UID: \"66ca18b4-5345-4c49-9772-e4bfd1bc6747\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:00:38.604542 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:38.604039    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:00:38.604542 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:38.604193    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 01:00:38.604542 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:38.604207    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 14 01:00:38.604542 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:38.604216    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d59cbd55f-nfdch: secret "image-registry-tls" not found
Jul 14 01:00:38.604542 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:38.604281    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert podName:66ca18b4-5345-4c49-9772-e4bfd1bc6747 nodeName:}" failed. No retries permitted until 2026-07-14 01:01:10.604256695 +0000 UTC m=+100.168665731 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-jhd7p" (UID: "66ca18b4-5345-4c49-9772-e4bfd1bc6747") : secret "networking-console-plugin-cert" not found
Jul 14 01:00:38.604542 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:38.604300    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls podName:4f83ab17-7261-408c-aaad-f38a36de4213 nodeName:}" failed. No retries permitted until 2026-07-14 01:01:10.604290126 +0000 UTC m=+100.168699159 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls") pod "image-registry-6d59cbd55f-nfdch" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213") : secret "image-registry-tls" not found
Jul 14 01:00:38.704908 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:38.704867    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:00:38.705114 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:38.705072    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 01:00:38.705184 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:00:38.705154    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls podName:ef8fc61b-7241-4cec-bda8-ecfb2ed789f1 nodeName:}" failed. No retries permitted until 2026-07-14 01:01:10.705133053 +0000 UTC m=+100.269542109 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls") pod "dns-default-jjkx4" (UID: "ef8fc61b-7241-4cec-bda8-ecfb2ed789f1") : secret "dns-default-metrics-tls" not found
Jul 14 01:00:41.509492 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:41.509408    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:41.610731 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:41.610686    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:43.344281 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:43.344240    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6dnxf" event={"ID":"455ca3e1-0a9d-4b8e-b150-06ba86301ae5","Type":"ContainerStarted","Data":"ccfb2353dc4a96b80672cda52c1e9398d8b4d9da9c2874e701624bdaf01d2a1d"}
Jul 14 01:00:43.344739 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:43.344326    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 01:00:43.345662 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:43.345638    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q8j4f" event={"ID":"1a747595-65ad-41fe-a1f1-d0914d6eadc2","Type":"ContainerStarted","Data":"1b631a8f143f29674d9604c3511e583b7e98ea79edf543299b54632abf1d0466"}
Jul 14 01:00:43.362401 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:43.362344    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6dnxf" podStartSLOduration=66.785720629 podStartE2EDuration="1m12.362326368s" totalImagesPullingTime="5.576605739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:43.361253048 +0000 UTC m=+72.925662104" watchObservedRunningTime="2026-07-14 01:00:43.362326368 +0000 UTC m=+72.926735422"
Jul 14 01:00:43.374483 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:43.374427    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q8j4f" podStartSLOduration=65.782957148 podStartE2EDuration="1m11.374408113s" totalImagesPullingTime="5.591450965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:43.373806334 +0000 UTC m=+72.938215391" watchObservedRunningTime="2026-07-14 01:00:43.374408113 +0000 UTC m=+72.938817168"
Jul 14 01:00:45.129078 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:45.129036    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:45.485706 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:45.485614    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:45.542673 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:45.542634    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:46.235479 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:46.235443    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:46.288326 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:46.288291    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:48.932455 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:48.932423    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:49.032416 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:49.032369    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:49.378450 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:49.378409    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:00:49.431583 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:00:49.431556    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:01:06.465447 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:01:06.465308    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:01:06.930550 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:01:06.930515    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:01:10.635159 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:01:10.635115    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jhd7p\" (UID: \"66ca18b4-5345-4c49-9772-e4bfd1bc6747\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:01:10.635563 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:01:10.635168    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:01:10.635563 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:01:10.635289    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 14 01:01:10.635563 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:01:10.635292    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 01:01:10.635563 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:01:10.635312    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d59cbd55f-nfdch: secret "image-registry-tls" not found
Jul 14 01:01:10.635563 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:01:10.635365    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls podName:4f83ab17-7261-408c-aaad-f38a36de4213 nodeName:}" failed. No retries permitted until 2026-07-14 01:02:14.635348357 +0000 UTC m=+164.199757390 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls") pod "image-registry-6d59cbd55f-nfdch" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213") : secret "image-registry-tls" not found
Jul 14 01:01:10.635563 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:01:10.635382    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert podName:66ca18b4-5345-4c49-9772-e4bfd1bc6747 nodeName:}" failed. No retries permitted until 2026-07-14 01:02:14.635372266 +0000 UTC m=+164.199781299 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-jhd7p" (UID: "66ca18b4-5345-4c49-9772-e4bfd1bc6747") : secret "networking-console-plugin-cert" not found
Jul 14 01:01:10.735646 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:01:10.735605    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:01:10.735789 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:01:10.735730    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 01:01:10.735789 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:01:10.735787    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls podName:ef8fc61b-7241-4cec-bda8-ecfb2ed789f1 nodeName:}" failed. No retries permitted until 2026-07-14 01:02:14.735770736 +0000 UTC m=+164.300179770 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls") pod "dns-default-jjkx4" (UID: "ef8fc61b-7241-4cec-bda8-ecfb2ed789f1") : secret "dns-default-metrics-tls" not found
Jul 14 01:01:11.675526 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:01:11.675490    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:01:11.729096 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:01:11.729066    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:01:14.350765 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:01:14.350731    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6dnxf"
Jul 14 01:01:40.753757 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:01:40.753721    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 01:01:40.754239 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:01:40.753855    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 14 01:01:40.754239 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:01:40.753926    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs podName:f0ebac4e-8f5b-4820-b018-8386d03067f1 nodeName:}" failed. No retries permitted until 2026-07-14 01:03:42.75390944 +0000 UTC m=+252.318318473 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs") pod "network-metrics-daemon-jfg2f" (UID: "f0ebac4e-8f5b-4820-b018-8386d03067f1") : secret "metrics-daemon-secret" not found
Jul 14 01:02:02.649761 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.649724    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"]
Jul 14 01:02:02.652038 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.652018    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"
Jul 14 01:02:02.654564 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.654537    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 14 01:02:02.654564 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.654558    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 14 01:02:02.655003 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.654983    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 14 01:02:02.655310 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.655055    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 14 01:02:02.661760 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.661738    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"]
Jul 14 01:02:02.709603 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.709554    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7e6250d-4439-4590-9d6d-f6261b4fa11d-tmp\") pod \"klusterlet-addon-workmgr-84bd75b5fc-6xb5h\" (UID: \"d7e6250d-4439-4590-9d6d-f6261b4fa11d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"
Jul 14 01:02:02.709776 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.709641    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d7e6250d-4439-4590-9d6d-f6261b4fa11d-klusterlet-config\") pod \"klusterlet-addon-workmgr-84bd75b5fc-6xb5h\" (UID: \"d7e6250d-4439-4590-9d6d-f6261b4fa11d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"
Jul 14 01:02:02.709776 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.709703    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qfp\" (UniqueName: \"kubernetes.io/projected/d7e6250d-4439-4590-9d6d-f6261b4fa11d-kube-api-access-z5qfp\") pod \"klusterlet-addon-workmgr-84bd75b5fc-6xb5h\" (UID: \"d7e6250d-4439-4590-9d6d-f6261b4fa11d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"
Jul 14 01:02:02.810847 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.810807    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7e6250d-4439-4590-9d6d-f6261b4fa11d-tmp\") pod \"klusterlet-addon-workmgr-84bd75b5fc-6xb5h\" (UID: \"d7e6250d-4439-4590-9d6d-f6261b4fa11d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"
Jul 14 01:02:02.810931 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.810886    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d7e6250d-4439-4590-9d6d-f6261b4fa11d-klusterlet-config\") pod \"klusterlet-addon-workmgr-84bd75b5fc-6xb5h\" (UID: \"d7e6250d-4439-4590-9d6d-f6261b4fa11d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"
Jul 14 01:02:02.810931 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.810912    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qfp\" (UniqueName: \"kubernetes.io/projected/d7e6250d-4439-4590-9d6d-f6261b4fa11d-kube-api-access-z5qfp\") pod \"klusterlet-addon-workmgr-84bd75b5fc-6xb5h\" (UID: \"d7e6250d-4439-4590-9d6d-f6261b4fa11d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"
Jul 14 01:02:02.811287 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.811256    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7e6250d-4439-4590-9d6d-f6261b4fa11d-tmp\") pod \"klusterlet-addon-workmgr-84bd75b5fc-6xb5h\" (UID: \"d7e6250d-4439-4590-9d6d-f6261b4fa11d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"
Jul 14 01:02:02.813641 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.813606    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d7e6250d-4439-4590-9d6d-f6261b4fa11d-klusterlet-config\") pod \"klusterlet-addon-workmgr-84bd75b5fc-6xb5h\" (UID: \"d7e6250d-4439-4590-9d6d-f6261b4fa11d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"
Jul 14 01:02:02.827196 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.827172    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qfp\" (UniqueName: \"kubernetes.io/projected/d7e6250d-4439-4590-9d6d-f6261b4fa11d-kube-api-access-z5qfp\") pod \"klusterlet-addon-workmgr-84bd75b5fc-6xb5h\" (UID: \"d7e6250d-4439-4590-9d6d-f6261b4fa11d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"
Jul 14 01:02:02.963859 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:02.963756    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"
Jul 14 01:02:03.087999 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:03.087945    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"]
Jul 14 01:02:03.091153 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:02:03.091122    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e6250d_4439_4590_9d6d_f6261b4fa11d.slice/crio-f97bf1e8892c47abf3737637eaa371a16bd0e793a78bb491a05a9e77274d2820 WatchSource:0}: Error finding container f97bf1e8892c47abf3737637eaa371a16bd0e793a78bb491a05a9e77274d2820: Status 404 returned error can't find the container with id f97bf1e8892c47abf3737637eaa371a16bd0e793a78bb491a05a9e77274d2820
Jul 14 01:02:03.533222 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:03.533178    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h" event={"ID":"d7e6250d-4439-4590-9d6d-f6261b4fa11d","Type":"ContainerStarted","Data":"f97bf1e8892c47abf3737637eaa371a16bd0e793a78bb491a05a9e77274d2820"}
Jul 14 01:02:09.757717 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:02:09.757663    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch" podUID="4f83ab17-7261-408c-aaad-f38a36de4213"
Jul 14 01:02:09.765040 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:02:09.764988    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p" podUID="66ca18b4-5345-4c49-9772-e4bfd1bc6747"
Jul 14 01:02:09.787307 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:02:09.787256    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jjkx4" podUID="ef8fc61b-7241-4cec-bda8-ecfb2ed789f1"
Jul 14 01:02:09.993131 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:02:09.993089    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 01:02:10.549617 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:10.549584    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:02:10.549617 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:10.549609    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:02:10.549871 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:10.549738    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:02:14.560874 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:14.560838    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h" event={"ID":"d7e6250d-4439-4590-9d6d-f6261b4fa11d","Type":"ContainerStarted","Data":"1647eb955c4a52e0bef4ce244de7a5a2454a97e21e237887c46503739cc5fd16"}
Jul 14 01:02:14.561292 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:14.561085    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"
Jul 14 01:02:14.562671 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:14.562651    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"
Jul 14 01:02:14.575147 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:14.575094    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h" podStartSLOduration=1.3317666479999999 podStartE2EDuration="12.575077945s" totalImagesPullingTime="11.243311297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:02:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:02:14.573908907 +0000 UTC m=+164.138317962" watchObservedRunningTime="2026-07-14 01:02:14.575077945 +0000 UTC m=+164.139486999"
Jul 14 01:02:14.699624 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:14.699580    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jhd7p\" (UID: \"66ca18b4-5345-4c49-9772-e4bfd1bc6747\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:02:14.699822 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:14.699633    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:02:14.699822 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:02:14.699735    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 01:02:14.699822 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:02:14.699745    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d59cbd55f-nfdch: secret "image-registry-tls" not found
Jul 14 01:02:14.699822 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:02:14.699743    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 14 01:02:14.699822 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:02:14.699794    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls podName:4f83ab17-7261-408c-aaad-f38a36de4213 nodeName:}" failed. No retries permitted until 2026-07-14 01:04:16.699780113 +0000 UTC m=+286.264189146 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls") pod "image-registry-6d59cbd55f-nfdch" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213") : secret "image-registry-tls" not found
Jul 14 01:02:14.699822 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:02:14.699806    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert podName:66ca18b4-5345-4c49-9772-e4bfd1bc6747 nodeName:}" failed. No retries permitted until 2026-07-14 01:04:16.699800527 +0000 UTC m=+286.264209560 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-jhd7p" (UID: "66ca18b4-5345-4c49-9772-e4bfd1bc6747") : secret "networking-console-plugin-cert" not found
Jul 14 01:02:14.800563 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:14.800520    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:02:14.800716 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:02:14.800649    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 01:02:14.800716 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:02:14.800699    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls podName:ef8fc61b-7241-4cec-bda8-ecfb2ed789f1 nodeName:}" failed. No retries permitted until 2026-07-14 01:04:16.800686502 +0000 UTC m=+286.365095536 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls") pod "dns-default-jjkx4" (UID: "ef8fc61b-7241-4cec-bda8-ecfb2ed789f1") : secret "dns-default-metrics-tls" not found
Jul 14 01:02:24.968743 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:02:24.968648    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 01:03:21.966900 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:21.966869    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:42.825714 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:42.825673    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 01:03:42.826234 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:42.825851    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 14 01:03:42.826234 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:42.825977    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs podName:f0ebac4e-8f5b-4820-b018-8386d03067f1 nodeName:}" failed. No retries permitted until 2026-07-14 01:05:44.825928658 +0000 UTC m=+374.390337699 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs") pod "network-metrics-daemon-jfg2f" (UID: "f0ebac4e-8f5b-4820-b018-8386d03067f1") : secret "metrics-daemon-secret" not found
Jul 14 01:03:47.602062 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.602027    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-cppnk"]
Jul 14 01:03:47.603829 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.603811    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cppnk"
Jul 14 01:03:47.606093 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.606070    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mfdxs\""
Jul 14 01:03:47.621937 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.621908    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-cppnk"]
Jul 14 01:03:47.657440 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.657400    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45qc6\" (UniqueName: \"kubernetes.io/projected/3e5cd9e2-9bd2-4e06-98b6-7735913b5d20-kube-api-access-45qc6\") pod \"network-check-source-869dcfcdcd-cppnk\" (UID: \"3e5cd9e2-9bd2-4e06-98b6-7735913b5d20\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cppnk"
Jul 14 01:03:47.709869 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.709841    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6c77944dd8-qlg7c"]
Jul 14 01:03:47.711732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.711709    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:47.713883 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.713850    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 14 01:03:47.714054 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.713913    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 14 01:03:47.714054 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.713920    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 01:03:47.714330 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.714283    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 01:03:47.714410 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.714334    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 14 01:03:47.714680 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.714563    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hpspl\""
Jul 14 01:03:47.714812 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.714793    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 14 01:03:47.724356 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.724333    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c77944dd8-qlg7c"]
Jul 14 01:03:47.758232 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.758189    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45qc6\" (UniqueName: \"kubernetes.io/projected/3e5cd9e2-9bd2-4e06-98b6-7735913b5d20-kube-api-access-45qc6\") pod \"network-check-source-869dcfcdcd-cppnk\" (UID: \"3e5cd9e2-9bd2-4e06-98b6-7735913b5d20\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cppnk"
Jul 14 01:03:47.758232 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.758232    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-default-certificate\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:47.758453 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.758250    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q6tm\" (UniqueName: \"kubernetes.io/projected/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-kube-api-access-4q6tm\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:47.758453 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.758273    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:47.758453 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.758296    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-stats-auth\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:47.758453 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.758317    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:47.768420 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.768395    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45qc6\" (UniqueName: \"kubernetes.io/projected/3e5cd9e2-9bd2-4e06-98b6-7735913b5d20-kube-api-access-45qc6\") pod \"network-check-source-869dcfcdcd-cppnk\" (UID: \"3e5cd9e2-9bd2-4e06-98b6-7735913b5d20\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cppnk"
Jul 14 01:03:47.859474 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.859380    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-default-certificate\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:47.859474 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.859414    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q6tm\" (UniqueName: \"kubernetes.io/projected/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-kube-api-access-4q6tm\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:47.859474 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.859440    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:47.859753 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.859481    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-stats-auth\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:47.859753 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.859504    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:47.859753 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:47.859606    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 14 01:03:47.859753 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:47.859675    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs podName:a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b nodeName:}" failed. No retries permitted until 2026-07-14 01:03:48.359657313 +0000 UTC m=+257.924066346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs") pod "router-default-6c77944dd8-qlg7c" (UID: "a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b") : secret "router-metrics-certs-default" not found
Jul 14 01:03:47.859753 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:47.859738    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle podName:a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b nodeName:}" failed. No retries permitted until 2026-07-14 01:03:48.359709186 +0000 UTC m=+257.924118253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle") pod "router-default-6c77944dd8-qlg7c" (UID: "a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b") : configmap references non-existent config key: service-ca.crt
Jul 14 01:03:47.862054 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.862024    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-stats-auth\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:47.862054 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.862032    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-default-certificate\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:47.869944 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.869914    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q6tm\" (UniqueName: \"kubernetes.io/projected/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-kube-api-access-4q6tm\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:47.912843 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.912804    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cppnk"
Jul 14 01:03:47.942248 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.942217    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:47.991151 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:47.991121    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:48.040588 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:48.040550    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-cppnk"]
Jul 14 01:03:48.043497 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:03:48.043469    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5cd9e2_9bd2_4e06_98b6_7735913b5d20.slice/crio-6aab61465b8bffd53010ccd2b36d85b4ea1999e745e32dfbbc419470657b6b35 WatchSource:0}: Error finding container 6aab61465b8bffd53010ccd2b36d85b4ea1999e745e32dfbbc419470657b6b35: Status 404 returned error can't find the container with id 6aab61465b8bffd53010ccd2b36d85b4ea1999e745e32dfbbc419470657b6b35
Jul 14 01:03:48.364184 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:48.364138    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:48.364184 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:48.364182    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:48.364392 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:48.364290    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 14 01:03:48.364392 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:48.364318    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle podName:a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b nodeName:}" failed. No retries permitted until 2026-07-14 01:03:49.364299625 +0000 UTC m=+258.928708657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle") pod "router-default-6c77944dd8-qlg7c" (UID: "a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b") : configmap references non-existent config key: service-ca.crt
Jul 14 01:03:48.364392 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:48.364341    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs podName:a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b nodeName:}" failed. No retries permitted until 2026-07-14 01:03:49.364334683 +0000 UTC m=+258.928743716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs") pod "router-default-6c77944dd8-qlg7c" (UID: "a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b") : secret "router-metrics-certs-default" not found
Jul 14 01:03:48.789829 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:48.789792    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cppnk" event={"ID":"3e5cd9e2-9bd2-4e06-98b6-7735913b5d20","Type":"ContainerStarted","Data":"1e5352ba345db76e74c6c4790da7befef5e9576465a02f79276b4a6f827ef3f1"}
Jul 14 01:03:48.790323 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:48.789851    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cppnk" event={"ID":"3e5cd9e2-9bd2-4e06-98b6-7735913b5d20","Type":"ContainerStarted","Data":"6aab61465b8bffd53010ccd2b36d85b4ea1999e745e32dfbbc419470657b6b35"}
Jul 14 01:03:48.803968 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:48.803904    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cppnk" podStartSLOduration=1.803889507 podStartE2EDuration="1.803889507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:03:48.803000845 +0000 UTC m=+258.367409902" watchObservedRunningTime="2026-07-14 01:03:48.803889507 +0000 UTC m=+258.368298562"
Jul 14 01:03:49.373242 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:49.373193    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:49.373242 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:49.373238    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:49.373522 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:49.373335    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 14 01:03:49.373522 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:49.373385    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs podName:a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b nodeName:}" failed. No retries permitted until 2026-07-14 01:03:51.37337024 +0000 UTC m=+260.937779273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs") pod "router-default-6c77944dd8-qlg7c" (UID: "a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b") : secret "router-metrics-certs-default" not found
Jul 14 01:03:49.373522 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:49.373469    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle podName:a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b nodeName:}" failed. No retries permitted until 2026-07-14 01:03:51.373462273 +0000 UTC m=+260.937871306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle") pod "router-default-6c77944dd8-qlg7c" (UID: "a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b") : configmap references non-existent config key: service-ca.crt
Jul 14 01:03:50.293036 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:50.293000    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:50.341702 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:50.341676    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:51.386724 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:51.386666    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:51.386724 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:51.386721    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:51.387213 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:51.386812    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 14 01:03:51.387213 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:51.386839    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle podName:a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b nodeName:}" failed. No retries permitted until 2026-07-14 01:03:55.386820499 +0000 UTC m=+264.951229532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle") pod "router-default-6c77944dd8-qlg7c" (UID: "a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b") : configmap references non-existent config key: service-ca.crt
Jul 14 01:03:51.387213 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:51.386865    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs podName:a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b nodeName:}" failed. No retries permitted until 2026-07-14 01:03:55.386858583 +0000 UTC m=+264.951267616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs") pod "router-default-6c77944dd8-qlg7c" (UID: "a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b") : secret "router-metrics-certs-default" not found
Jul 14 01:03:52.004266 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:52.004234    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:52.055013 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:52.054988    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:52.499010 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:52.498975    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:52.602493 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:52.602463    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:52.875048 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:52.875015    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hws8z"]
Jul 14 01:03:52.877019 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:52.876997    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:52.878791 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:52.878759    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 14 01:03:52.878913 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:52.878765    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 14 01:03:52.878913 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:52.878832    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 14 01:03:52.878913 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:52.878842    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 14 01:03:52.879331 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:52.879318    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vfczl\""
Jul 14 01:03:52.889204 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:52.889180    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hws8z"]
Jul 14 01:03:52.907514 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:52.907484    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:52.963034 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:52.962999    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:53.002176 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:53.002134    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2w5m\" (UniqueName: \"kubernetes.io/projected/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-api-access-d2w5m\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:53.002176 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:53.002182    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c4e0e77-c547-406c-b68e-0bfff407fcf2-crio-socket\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:53.002395 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:53.002225    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:53.002395 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:53.002302    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:53.002395 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:53.002335    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0e77-c547-406c-b68e-0bfff407fcf2-data-volume\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:53.103488 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:53.103453    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2w5m\" (UniqueName: \"kubernetes.io/projected/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-api-access-d2w5m\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:53.103582 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:53.103501    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c4e0e77-c547-406c-b68e-0bfff407fcf2-crio-socket\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:53.103582 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:53.103569    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c4e0e77-c547-406c-b68e-0bfff407fcf2-crio-socket\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:53.103661 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:53.103606    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:53.103661 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:53.103639    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:53.103736 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:53.103661    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0e77-c547-406c-b68e-0bfff407fcf2-data-volume\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:53.103772 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:53.103733    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 01:03:53.103815 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:53.103778    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 01:03:53.103815 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:53.103807    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm podName:2c4e0e77-c547-406c-b68e-0bfff407fcf2 nodeName:}" failed. No retries permitted until 2026-07-14 01:03:53.603787259 +0000 UTC m=+263.168196292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hws8z" (UID: "2c4e0e77-c547-406c-b68e-0bfff407fcf2") : configmap "kube-rbac-proxy" not found
Jul 14 01:03:53.103897 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:53.103830    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls podName:2c4e0e77-c547-406c-b68e-0bfff407fcf2 nodeName:}" failed. No retries permitted until 2026-07-14 01:03:53.603817647 +0000 UTC m=+263.168226681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hws8z" (UID: "2c4e0e77-c547-406c-b68e-0bfff407fcf2") : secret "insights-runtime-extractor-tls" not found
Jul 14 01:03:53.103990 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:53.103947    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c4e0e77-c547-406c-b68e-0bfff407fcf2-data-volume\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:53.111351 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:53.111327    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2w5m\" (UniqueName: \"kubernetes.io/projected/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-api-access-d2w5m\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:53.606513 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:53.606467    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:53.606887 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:53.606579    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:53.606887 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:53.606618    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 01:03:53.606887 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:53.606646    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 01:03:53.606887 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:53.606686    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls podName:2c4e0e77-c547-406c-b68e-0bfff407fcf2 nodeName:}" failed. No retries permitted until 2026-07-14 01:03:54.606671942 +0000 UTC m=+264.171080975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hws8z" (UID: "2c4e0e77-c547-406c-b68e-0bfff407fcf2") : secret "insights-runtime-extractor-tls" not found
Jul 14 01:03:53.606887 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:53.606701    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm podName:2c4e0e77-c547-406c-b68e-0bfff407fcf2 nodeName:}" failed. No retries permitted until 2026-07-14 01:03:54.606694596 +0000 UTC m=+264.171103630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hws8z" (UID: "2c4e0e77-c547-406c-b68e-0bfff407fcf2") : configmap "kube-rbac-proxy" not found
Jul 14 01:03:54.157130 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:54.157099    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:54.206073 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:54.206036    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:54.614900 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:54.614809    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:54.614900 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:54.614859    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:54.615326 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:54.614938    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 01:03:54.615326 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:54.615021    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 01:03:54.615326 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:54.615026    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm podName:2c4e0e77-c547-406c-b68e-0bfff407fcf2 nodeName:}" failed. No retries permitted until 2026-07-14 01:03:56.615006543 +0000 UTC m=+266.179415577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hws8z" (UID: "2c4e0e77-c547-406c-b68e-0bfff407fcf2") : configmap "kube-rbac-proxy" not found
Jul 14 01:03:54.615326 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:54.615062    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls podName:2c4e0e77-c547-406c-b68e-0bfff407fcf2 nodeName:}" failed. No retries permitted until 2026-07-14 01:03:56.61505258 +0000 UTC m=+266.179461614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hws8z" (UID: "2c4e0e77-c547-406c-b68e-0bfff407fcf2") : secret "insights-runtime-extractor-tls" not found
Jul 14 01:03:54.669555 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:54.669530    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d4fkp_2998116f-cba6-4d84-a68c-c4da115286ad/dns-node-resolver/0.log"
Jul 14 01:03:54.961458 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:54.961422    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:55.423059 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:55.423018    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:55.423059 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:55.423067    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:03:55.423325 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:55.423167    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 14 01:03:55.423325 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:55.423224    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle podName:a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b nodeName:}" failed. No retries permitted until 2026-07-14 01:04:03.423201952 +0000 UTC m=+272.987610986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle") pod "router-default-6c77944dd8-qlg7c" (UID: "a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b") : configmap references non-existent config key: service-ca.crt
Jul 14 01:03:55.423325 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:55.423252    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs podName:a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b nodeName:}" failed. No retries permitted until 2026-07-14 01:04:03.423242389 +0000 UTC m=+272.987651422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs") pod "router-default-6c77944dd8-qlg7c" (UID: "a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b") : secret "router-metrics-certs-default" not found
Jul 14 01:03:55.868335 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:55.868308    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jpnj7_b5459bba-8c2e-4be1-802f-a462a23327f6/node-ca/0.log"
Jul 14 01:03:56.633059 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:56.633027    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:56.633246 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:56.633073    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:03:56.633246 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:56.633148    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 01:03:56.633246 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:56.633243    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm podName:2c4e0e77-c547-406c-b68e-0bfff407fcf2 nodeName:}" failed. No retries permitted until 2026-07-14 01:04:00.633220375 +0000 UTC m=+270.197629428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hws8z" (UID: "2c4e0e77-c547-406c-b68e-0bfff407fcf2") : configmap "kube-rbac-proxy" not found
Jul 14 01:03:56.633356 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:56.633245    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 01:03:56.633356 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:03:56.633299    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls podName:2c4e0e77-c547-406c-b68e-0bfff407fcf2 nodeName:}" failed. No retries permitted until 2026-07-14 01:04:00.633287223 +0000 UTC m=+270.197696256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hws8z" (UID: "2c4e0e77-c547-406c-b68e-0bfff407fcf2") : secret "insights-runtime-extractor-tls" not found
Jul 14 01:03:56.641209 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:56.641177    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:56.671679 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:56.671651    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ghmc7_c8f406c4-b03c-48ff-a9d0-d12af1c70dff/serve-healthcheck-canary/0.log"
Jul 14 01:03:56.790134 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:56.790097    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:57.027267 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:57.027179    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:57.139786 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:57.139754    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:57.731867 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:57.731832    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:58.078871 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:58.078786    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:58.180037 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:58.180006    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:58.533134 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:58.533102    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:58.947054 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:58.947016    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:59.126828 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:59.126791    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:03:59.532503 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:03:59.532471    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:00.134270 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:00.134236    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:00.523887 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:00.523801    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:00.624790 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:00.624758    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:00.665668 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:00.665631    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:04:00.665824 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:00.665677    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:04:00.665824 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:04:00.665745    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 01:04:00.665824 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:04:00.665769    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 01:04:00.665824 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:04:00.665805    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm podName:2c4e0e77-c547-406c-b68e-0bfff407fcf2 nodeName:}" failed. No retries permitted until 2026-07-14 01:04:08.665791209 +0000 UTC m=+278.230200242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hws8z" (UID: "2c4e0e77-c547-406c-b68e-0bfff407fcf2") : configmap "kube-rbac-proxy" not found
Jul 14 01:04:00.665824 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:04:00.665818    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls podName:2c4e0e77-c547-406c-b68e-0bfff407fcf2 nodeName:}" failed. No retries permitted until 2026-07-14 01:04:08.665812732 +0000 UTC m=+278.230221765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hws8z" (UID: "2c4e0e77-c547-406c-b68e-0bfff407fcf2") : secret "insights-runtime-extractor-tls" not found
Jul 14 01:04:00.929593 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:00.929562    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:01.132466 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:01.132434    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:01.402634 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:01.402606    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:01.657496 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:01.657416    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:02.349060 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:02.349025    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:02.752025 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:02.751994    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:02.926813 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:02.926768    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:03.099562 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:03.099476    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:03.153674 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:03.153645    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:03.329511 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:03.329477    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:03.488940 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:03.488892    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:04:03.488940 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:03.488944    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:04:03.489552 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:03.489534    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-service-ca-bundle\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:04:03.491373 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:03.491354    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b-metrics-certs\") pod \"router-default-6c77944dd8-qlg7c\" (UID: \"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b\") " pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:04:03.620915 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:03.620883    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:04:03.749643 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:03.749558    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6c77944dd8-qlg7c"]
Jul 14 01:04:03.752921 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:04:03.752896    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cab32d_5e49_4a6e_b5e4_f5bbfa7e7e4b.slice/crio-df080132b56dc1de9c4bf6cfeadf575f72d77146c5f1ac7f5b993286cf140e11 WatchSource:0}: Error finding container df080132b56dc1de9c4bf6cfeadf575f72d77146c5f1ac7f5b993286cf140e11: Status 404 returned error can't find the container with id df080132b56dc1de9c4bf6cfeadf575f72d77146c5f1ac7f5b993286cf140e11
Jul 14 01:04:03.829473 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:03.829439    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c77944dd8-qlg7c" event={"ID":"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b","Type":"ContainerStarted","Data":"a409bec59d7444c8d321e2e5064a5d0cb676cee5510288792ce91ff61142ce87"}
Jul 14 01:04:03.829675 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:03.829495    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6c77944dd8-qlg7c" event={"ID":"a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b","Type":"ContainerStarted","Data":"df080132b56dc1de9c4bf6cfeadf575f72d77146c5f1ac7f5b993286cf140e11"}
Jul 14 01:04:03.848986 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:03.848912    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6c77944dd8-qlg7c" podStartSLOduration=16.848888575 podStartE2EDuration="16.848888575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:04:03.848132012 +0000 UTC m=+273.412541066" watchObservedRunningTime="2026-07-14 01:04:03.848888575 +0000 UTC m=+273.413297650"
Jul 14 01:04:03.928452 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:03.928415    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:04.621064 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:04.621027    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:04:04.623638 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:04.623613    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:04:04.794892 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:04.794848    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:04.832500 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:04.832461    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:04:04.833794 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:04.833770    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6c77944dd8-qlg7c"
Jul 14 01:04:05.051744 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:05.051659    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:05.153420 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:05.153384    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:05.332330 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:05.332237    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:05.525764 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:05.525730    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:05.726025 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:05.725989    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:06.537834 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:06.537789    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:06.727779 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:06.727747    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:06.944417 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:06.944356    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:07.044119 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:07.044065    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:07.143605 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:07.143560    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:07.335411 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:07.335314    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:08.736172 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:08.736129    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:04:08.736172 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:08.736178    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:04:08.736671 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:08.736633    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c4e0e77-c547-406c-b68e-0bfff407fcf2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:04:08.738623 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:08.738600    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c4e0e77-c547-406c-b68e-0bfff407fcf2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hws8z\" (UID: \"2c4e0e77-c547-406c-b68e-0bfff407fcf2\") " pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:04:08.786210 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:08.786168    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hws8z"
Jul 14 01:04:08.915807 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:08.915775    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hws8z"]
Jul 14 01:04:08.918988 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:04:08.918943    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4e0e77_c547_406c_b68e_0bfff407fcf2.slice/crio-deaca288d4048ee7ebf783e8102d9dc9790bbc442ab97621dfe684722dadb9fc WatchSource:0}: Error finding container deaca288d4048ee7ebf783e8102d9dc9790bbc442ab97621dfe684722dadb9fc: Status 404 returned error can't find the container with id deaca288d4048ee7ebf783e8102d9dc9790bbc442ab97621dfe684722dadb9fc
Jul 14 01:04:09.851915 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:09.851876    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hws8z" event={"ID":"2c4e0e77-c547-406c-b68e-0bfff407fcf2","Type":"ContainerStarted","Data":"deaca288d4048ee7ebf783e8102d9dc9790bbc442ab97621dfe684722dadb9fc"}
Jul 14 01:04:12.863727 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:12.863691    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hws8z" event={"ID":"2c4e0e77-c547-406c-b68e-0bfff407fcf2","Type":"ContainerStarted","Data":"b4ae09f105bc85cee880be5f8aefa72e2199d2729a34b1e8af20aeb31673efed"}
Jul 14 01:04:13.551040 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:04:13.550992    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch" podUID="4f83ab17-7261-408c-aaad-f38a36de4213"
Jul 14 01:04:13.551040 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:04:13.551010    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-jjkx4" podUID="ef8fc61b-7241-4cec-bda8-ecfb2ed789f1"
Jul 14 01:04:13.551040 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:04:13.551010    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p" podUID="66ca18b4-5345-4c49-9772-e4bfd1bc6747"
Jul 14 01:04:13.625556 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:13.625481    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:13.674064 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:13.674033    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:13.868392 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:13.868354    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:04:13.868832 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:13.868397    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hws8z" event={"ID":"2c4e0e77-c547-406c-b68e-0bfff407fcf2","Type":"ContainerStarted","Data":"aa1323b6ba240ea26fbe6614a065d0f9f5b1cee4a55c780668c1056808a991cb"}
Jul 14 01:04:13.868832 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:13.868462    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:04:13.868832 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:13.868593    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:04:14.252724 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.252504    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-ljjxb"]
Jul 14 01:04:14.255195 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.255165    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-ljjxb"
Jul 14 01:04:14.257364 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.257131    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 14 01:04:14.257686 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.257664    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 14 01:04:14.257796 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.257670    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-m879l\""
Jul 14 01:04:14.266321 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.266292    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-ljjxb"]
Jul 14 01:04:14.305171 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.305135    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:14.387351 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.387307    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktr49\" (UniqueName: \"kubernetes.io/projected/6e840f36-bcc1-473f-8f2d-36de1ebc376c-kube-api-access-ktr49\") pod \"migrator-7f59b54f6c-ljjxb\" (UID: \"6e840f36-bcc1-473f-8f2d-36de1ebc376c\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-ljjxb"
Jul 14 01:04:14.408679 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.408644    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:14.488363 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.488276    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktr49\" (UniqueName: \"kubernetes.io/projected/6e840f36-bcc1-473f-8f2d-36de1ebc376c-kube-api-access-ktr49\") pod \"migrator-7f59b54f6c-ljjxb\" (UID: \"6e840f36-bcc1-473f-8f2d-36de1ebc376c\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-ljjxb"
Jul 14 01:04:14.496100 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.496062    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktr49\" (UniqueName: \"kubernetes.io/projected/6e840f36-bcc1-473f-8f2d-36de1ebc376c-kube-api-access-ktr49\") pod \"migrator-7f59b54f6c-ljjxb\" (UID: \"6e840f36-bcc1-473f-8f2d-36de1ebc376c\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-ljjxb"
Jul 14 01:04:14.500736 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.500704    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:14.562380 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.562330    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h" podUID="d7e6250d-4439-4590-9d6d-f6261b4fa11d" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.10:8000/readyz\": dial tcp 10.132.0.10:8000: connect: connection refused"
Jul 14 01:04:14.567679 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.567650    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-ljjxb"
Jul 14 01:04:14.714102 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.714067    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-ljjxb"]
Jul 14 01:04:14.718621 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:04:14.718576    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e840f36_bcc1_473f_8f2d_36de1ebc376c.slice/crio-51eb47ab4cfb7034486663f9c3d883ecf896bbd3e3a586eebe7340d1ce58e331 WatchSource:0}: Error finding container 51eb47ab4cfb7034486663f9c3d883ecf896bbd3e3a586eebe7340d1ce58e331: Status 404 returned error can't find the container with id 51eb47ab4cfb7034486663f9c3d883ecf896bbd3e3a586eebe7340d1ce58e331
Jul 14 01:04:14.873629 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.873591    2567 generic.go:356] "Generic (PLEG): container finished" podID="d7e6250d-4439-4590-9d6d-f6261b4fa11d" containerID="1647eb955c4a52e0bef4ce244de7a5a2454a97e21e237887c46503739cc5fd16" exitCode=1
Jul 14 01:04:14.874131 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.873689    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h" event={"ID":"d7e6250d-4439-4590-9d6d-f6261b4fa11d","Type":"ContainerDied","Data":"1647eb955c4a52e0bef4ce244de7a5a2454a97e21e237887c46503739cc5fd16"}
Jul 14 01:04:14.874194 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.874139    2567 scope.go:117] "RemoveContainer" containerID="1647eb955c4a52e0bef4ce244de7a5a2454a97e21e237887c46503739cc5fd16"
Jul 14 01:04:14.875092 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:14.875066    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-ljjxb" event={"ID":"6e840f36-bcc1-473f-8f2d-36de1ebc376c","Type":"ContainerStarted","Data":"51eb47ab4cfb7034486663f9c3d883ecf896bbd3e3a586eebe7340d1ce58e331"}
Jul 14 01:04:15.881016 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:15.880976    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h" event={"ID":"d7e6250d-4439-4590-9d6d-f6261b4fa11d","Type":"ContainerStarted","Data":"64da8a7e8759fc8761ddf3b3a530f8e35a4984a28b151149cae442d9374f0d2c"}
Jul 14 01:04:15.881478 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:15.881415    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"
Jul 14 01:04:15.882001 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:15.881981    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84bd75b5fc-6xb5h"
Jul 14 01:04:16.705876 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:16.705830    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jhd7p\" (UID: \"66ca18b4-5345-4c49-9772-e4bfd1bc6747\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:04:16.705876 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:16.705874    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:04:16.708486 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:16.708451    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/66ca18b4-5345-4c49-9772-e4bfd1bc6747-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-jhd7p\" (UID: \"66ca18b4-5345-4c49-9772-e4bfd1bc6747\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:04:16.708598 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:16.708461    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls\") pod \"image-registry-6d59cbd55f-nfdch\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") " pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:04:16.807274 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:16.807235    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:04:16.810010 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:16.809989    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef8fc61b-7241-4cec-bda8-ecfb2ed789f1-metrics-tls\") pod \"dns-default-jjkx4\" (UID: \"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1\") " pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:04:16.871817 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:16.871779    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jf2dk\""
Jul 14 01:04:16.871817 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:16.871812    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-sp2ms\""
Jul 14 01:04:16.872074 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:16.871842    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5xx4b\""
Jul 14 01:04:16.879486 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:16.879458    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"
Jul 14 01:04:16.879609 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:16.879489    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:04:16.879609 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:16.879575    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:04:16.885940 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:16.885891    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hws8z" event={"ID":"2c4e0e77-c547-406c-b68e-0bfff407fcf2","Type":"ContainerStarted","Data":"e4cc3eb91bfd9f5cee76691a57fbe90cbba4e1e9d3f428e998626602bd420589"}
Jul 14 01:04:16.904285 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:16.903594    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hws8z" podStartSLOduration=17.912459766 podStartE2EDuration="24.903568381s" totalImagesPullingTime="6.991108615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 01:03:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:04:16.901093057 +0000 UTC m=+286.465502113" watchObservedRunningTime="2026-07-14 01:04:16.903568381 +0000 UTC m=+286.467977437"
Jul 14 01:04:17.048014 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:17.047727    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d59cbd55f-nfdch"]
Jul 14 01:04:17.230208 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:04:17.230171    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f83ab17_7261_408c_aaad_f38a36de4213.slice/crio-cfb0ec41f960e91b3e6a1ff314420a7f7982b1d5963d6acbf70cb7d96cc4b8e1 WatchSource:0}: Error finding container cfb0ec41f960e91b3e6a1ff314420a7f7982b1d5963d6acbf70cb7d96cc4b8e1: Status 404 returned error can't find the container with id cfb0ec41f960e91b3e6a1ff314420a7f7982b1d5963d6acbf70cb7d96cc4b8e1
Jul 14 01:04:17.273342 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:17.273309    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-jhd7p"]
Jul 14 01:04:17.276249 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:17.276213    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jjkx4"]
Jul 14 01:04:17.283060 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:04:17.283032    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8fc61b_7241_4cec_bda8_ecfb2ed789f1.slice/crio-07bef07460a3a9faac081f3dec5bf4e3cebe964243208f58c7f57ed19611a486 WatchSource:0}: Error finding container 07bef07460a3a9faac081f3dec5bf4e3cebe964243208f58c7f57ed19611a486: Status 404 returned error can't find the container with id 07bef07460a3a9faac081f3dec5bf4e3cebe964243208f58c7f57ed19611a486
Jul 14 01:04:17.283240 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:04:17.283225    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ca18b4_5345_4c49_9772_e4bfd1bc6747.slice/crio-a51a194c08156e0c0a21ce2f934ec9696533b43e52a572962ed8d5115da7c151 WatchSource:0}: Error finding container a51a194c08156e0c0a21ce2f934ec9696533b43e52a572962ed8d5115da7c151: Status 404 returned error can't find the container with id a51a194c08156e0c0a21ce2f934ec9696533b43e52a572962ed8d5115da7c151
Jul 14 01:04:17.891604 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:17.891567    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch" event={"ID":"4f83ab17-7261-408c-aaad-f38a36de4213","Type":"ContainerStarted","Data":"49463c6a23016b02e7893e140c65dca2e70ff3b1d54a3ac45baa4fe321c2c2fe"}
Jul 14 01:04:17.892132 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:17.891624    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch" event={"ID":"4f83ab17-7261-408c-aaad-f38a36de4213","Type":"ContainerStarted","Data":"cfb0ec41f960e91b3e6a1ff314420a7f7982b1d5963d6acbf70cb7d96cc4b8e1"}
Jul 14 01:04:17.892132 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:17.891676    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:04:17.892907 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:17.892879    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p" event={"ID":"66ca18b4-5345-4c49-9772-e4bfd1bc6747","Type":"ContainerStarted","Data":"a51a194c08156e0c0a21ce2f934ec9696533b43e52a572962ed8d5115da7c151"}
Jul 14 01:04:17.894200 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:17.894170    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjkx4" event={"ID":"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1","Type":"ContainerStarted","Data":"07bef07460a3a9faac081f3dec5bf4e3cebe964243208f58c7f57ed19611a486"}
Jul 14 01:04:17.896241 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:17.896216    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-ljjxb" event={"ID":"6e840f36-bcc1-473f-8f2d-36de1ebc376c","Type":"ContainerStarted","Data":"148f663184f3516e98f0c94db55dcac0077a9045156a2b9683d071e55876f764"}
Jul 14 01:04:17.896344 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:17.896264    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-ljjxb" event={"ID":"6e840f36-bcc1-473f-8f2d-36de1ebc376c","Type":"ContainerStarted","Data":"a74245c279e1814a6e51aa31be2b3bbc7dc8c0156de13a6924c4a3ff22d02771"}
Jul 14 01:04:17.909657 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:17.909601    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch" podStartSLOduration=285.90958086 podStartE2EDuration="4m45.90958086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:04:17.908061004 +0000 UTC m=+287.472470059" watchObservedRunningTime="2026-07-14 01:04:17.90958086 +0000 UTC m=+287.473989912"
Jul 14 01:04:17.920761 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:17.920703    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-ljjxb" podStartSLOduration=1.385988139 podStartE2EDuration="3.920683902s" totalImagesPullingTime="2.534695763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:04:17.920174565 +0000 UTC m=+287.484583621" watchObservedRunningTime="2026-07-14 01:04:17.920683902 +0000 UTC m=+287.485092934"
Jul 14 01:04:17.996274 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:17.996243    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:18.096455 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:18.096420    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:18.902345 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:18.902253    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p" event={"ID":"66ca18b4-5345-4c49-9772-e4bfd1bc6747","Type":"ContainerStarted","Data":"ff7d2c1e4ca4e9853de78fe0dba6ec7d1f9216ea95f38279849d013e6ba24c74"}
Jul 14 01:04:18.916629 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:18.916558    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-jhd7p" podStartSLOduration=264.514921789 podStartE2EDuration="4m25.916536342s" totalImagesPullingTime="1.401614553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:59:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:04:18.915347851 +0000 UTC m=+288.479756907" watchObservedRunningTime="2026-07-14 01:04:18.916536342 +0000 UTC m=+288.480945396"
Jul 14 01:04:18.987185 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:18.987095    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:19.088703 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:19.088656    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:19.139917 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:19.139855    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:19.907352 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:19.907310    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjkx4" event={"ID":"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1","Type":"ContainerStarted","Data":"847c8d5c04d00da76f725589517a1cc7a7f96f22ba78d694f29ce7e6ecea1147"}
Jul 14 01:04:19.907815 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:19.907366    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjkx4" event={"ID":"ef8fc61b-7241-4cec-bda8-ecfb2ed789f1","Type":"ContainerStarted","Data":"6c3be870f114e5fa138891d318233a8222b1469bf34b46ea8afe57688e9222ac"}
Jul 14 01:04:19.907815 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:19.907397    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:04:19.921938 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:19.921890    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jjkx4" podStartSLOduration=251.970310439 podStartE2EDuration="4m13.921871877s" totalImagesPullingTime="1.951561438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:04:19.92078353 +0000 UTC m=+289.485192586" watchObservedRunningTime="2026-07-14 01:04:19.921871877 +0000 UTC m=+289.486280933"
Jul 14 01:04:21.048895 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:21.048860    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:21.102677 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:21.102626    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:22.282677 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:22.282639    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:22.424053 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:22.424013    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-m4tmv"]
Jul 14 01:04:22.427038 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:22.427020    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-m4tmv"
Jul 14 01:04:22.428795 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:22.428765    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 14 01:04:22.428920 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:22.428846    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pmbhx\""
Jul 14 01:04:22.434107 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:22.434083    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-m4tmv"]
Jul 14 01:04:22.551974 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:22.551851    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/77bd9659-0c61-44e0-bc53-0f2084fa0bb6-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-m4tmv\" (UID: \"77bd9659-0c61-44e0-bc53-0f2084fa0bb6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-m4tmv"
Jul 14 01:04:22.652409 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:22.652367    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/77bd9659-0c61-44e0-bc53-0f2084fa0bb6-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-m4tmv\" (UID: \"77bd9659-0c61-44e0-bc53-0f2084fa0bb6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-m4tmv"
Jul 14 01:04:22.652559 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:04:22.652538    2567 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 14 01:04:22.652615 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:04:22.652604    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77bd9659-0c61-44e0-bc53-0f2084fa0bb6-tls-certificates podName:77bd9659-0c61-44e0-bc53-0f2084fa0bb6 nodeName:}" failed. No retries permitted until 2026-07-14 01:04:23.152586062 +0000 UTC m=+292.716995117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/77bd9659-0c61-44e0-bc53-0f2084fa0bb6-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-m4tmv" (UID: "77bd9659-0c61-44e0-bc53-0f2084fa0bb6") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 14 01:04:23.025248 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:23.025215    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:23.075676 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:23.075648    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:23.156793 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:23.156756    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/77bd9659-0c61-44e0-bc53-0f2084fa0bb6-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-m4tmv\" (UID: \"77bd9659-0c61-44e0-bc53-0f2084fa0bb6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-m4tmv"
Jul 14 01:04:23.159487 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:23.159458    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/77bd9659-0c61-44e0-bc53-0f2084fa0bb6-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-m4tmv\" (UID: \"77bd9659-0c61-44e0-bc53-0f2084fa0bb6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-m4tmv"
Jul 14 01:04:23.336165 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:23.336063    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-m4tmv"
Jul 14 01:04:23.462914 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:23.462881    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-m4tmv"]
Jul 14 01:04:23.466279 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:04:23.466248    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bd9659_0c61_44e0_bc53_0f2084fa0bb6.slice/crio-c8addb1c19d7dc6397fea0b17a99c7aa280e6a8ed308c856d45ad92383d51196 WatchSource:0}: Error finding container c8addb1c19d7dc6397fea0b17a99c7aa280e6a8ed308c856d45ad92383d51196: Status 404 returned error can't find the container with id c8addb1c19d7dc6397fea0b17a99c7aa280e6a8ed308c856d45ad92383d51196
Jul 14 01:04:23.921366 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:23.921323    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-m4tmv" event={"ID":"77bd9659-0c61-44e0-bc53-0f2084fa0bb6","Type":"ContainerStarted","Data":"c8addb1c19d7dc6397fea0b17a99c7aa280e6a8ed308c856d45ad92383d51196"}
Jul 14 01:04:24.439271 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:24.439230    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:24.490460 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:24.490430    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:25.929625 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:25.929573    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-m4tmv" event={"ID":"77bd9659-0c61-44e0-bc53-0f2084fa0bb6","Type":"ContainerStarted","Data":"752bdbfa1f7f4ea5f8c3911485b5e4009dd0769b12c45e1517dc50b1e1cc9de2"}
Jul 14 01:04:25.930028 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:25.929646    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-m4tmv"
Jul 14 01:04:25.937825 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:25.937796    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-m4tmv"
Jul 14 01:04:25.943348 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:25.943303    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-m4tmv" podStartSLOduration=2.503830857 podStartE2EDuration="3.943287448s" totalImagesPullingTime="1.439456591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:04:25.94280767 +0000 UTC m=+295.507216726" watchObservedRunningTime="2026-07-14 01:04:25.943287448 +0000 UTC m=+295.507696503"
Jul 14 01:04:26.451556 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.451517    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:26.486857 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.486828    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-k24bj"]
Jul 14 01:04:26.490169 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.490151    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj"
Jul 14 01:04:26.492117 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.492090    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-h7rg5\""
Jul 14 01:04:26.492251 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.492090    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 14 01:04:26.492705 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.492678    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 14 01:04:26.492705 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.492688    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 14 01:04:26.492831 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.492696    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 14 01:04:26.492831 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.492723    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 14 01:04:26.497369 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.497342    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-k24bj"]
Jul 14 01:04:26.513365 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.513330    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:26.572207 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.572173    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:26.686053 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.686015    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f8lk\" (UniqueName: \"kubernetes.io/projected/32a8e14b-c44c-48c4-90af-656a3555d54e-kube-api-access-4f8lk\") pod \"prometheus-operator-64785b6675-k24bj\" (UID: \"32a8e14b-c44c-48c4-90af-656a3555d54e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj"
Jul 14 01:04:26.686228 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.686062    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32a8e14b-c44c-48c4-90af-656a3555d54e-metrics-client-ca\") pod \"prometheus-operator-64785b6675-k24bj\" (UID: \"32a8e14b-c44c-48c4-90af-656a3555d54e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj"
Jul 14 01:04:26.686228 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.686090    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32a8e14b-c44c-48c4-90af-656a3555d54e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-k24bj\" (UID: \"32a8e14b-c44c-48c4-90af-656a3555d54e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj"
Jul 14 01:04:26.686228 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.686134    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/32a8e14b-c44c-48c4-90af-656a3555d54e-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-k24bj\" (UID: \"32a8e14b-c44c-48c4-90af-656a3555d54e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj"
Jul 14 01:04:26.787130 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.787039    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f8lk\" (UniqueName: \"kubernetes.io/projected/32a8e14b-c44c-48c4-90af-656a3555d54e-kube-api-access-4f8lk\") pod \"prometheus-operator-64785b6675-k24bj\" (UID: \"32a8e14b-c44c-48c4-90af-656a3555d54e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj"
Jul 14 01:04:26.787130 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.787086    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32a8e14b-c44c-48c4-90af-656a3555d54e-metrics-client-ca\") pod \"prometheus-operator-64785b6675-k24bj\" (UID: \"32a8e14b-c44c-48c4-90af-656a3555d54e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj"
Jul 14 01:04:26.787130 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.787127    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32a8e14b-c44c-48c4-90af-656a3555d54e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-k24bj\" (UID: \"32a8e14b-c44c-48c4-90af-656a3555d54e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj"
Jul 14 01:04:26.787401 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.787175    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/32a8e14b-c44c-48c4-90af-656a3555d54e-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-k24bj\" (UID: \"32a8e14b-c44c-48c4-90af-656a3555d54e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj"
Jul 14 01:04:26.787853 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.787828    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32a8e14b-c44c-48c4-90af-656a3555d54e-metrics-client-ca\") pod \"prometheus-operator-64785b6675-k24bj\" (UID: \"32a8e14b-c44c-48c4-90af-656a3555d54e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj"
Jul 14 01:04:26.789712 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.789688    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32a8e14b-c44c-48c4-90af-656a3555d54e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-k24bj\" (UID: \"32a8e14b-c44c-48c4-90af-656a3555d54e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj"
Jul 14 01:04:26.789875 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.789852    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/32a8e14b-c44c-48c4-90af-656a3555d54e-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-k24bj\" (UID: \"32a8e14b-c44c-48c4-90af-656a3555d54e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj"
Jul 14 01:04:26.794521 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.794497    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f8lk\" (UniqueName: \"kubernetes.io/projected/32a8e14b-c44c-48c4-90af-656a3555d54e-kube-api-access-4f8lk\") pod \"prometheus-operator-64785b6675-k24bj\" (UID: \"32a8e14b-c44c-48c4-90af-656a3555d54e\") " pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj"
Jul 14 01:04:26.799375 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.799351    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj"
Jul 14 01:04:26.929135 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:26.929100    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-k24bj"]
Jul 14 01:04:26.933591 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:04:26.933563    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a8e14b_c44c_48c4_90af_656a3555d54e.slice/crio-105811332b0d2d3fe73962ed8031bec53331e351ca8abff3a492998abff72e5c WatchSource:0}: Error finding container 105811332b0d2d3fe73962ed8031bec53331e351ca8abff3a492998abff72e5c: Status 404 returned error can't find the container with id 105811332b0d2d3fe73962ed8031bec53331e351ca8abff3a492998abff72e5c
Jul 14 01:04:27.938016 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:27.937980    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj" event={"ID":"32a8e14b-c44c-48c4-90af-656a3555d54e","Type":"ContainerStarted","Data":"105811332b0d2d3fe73962ed8031bec53331e351ca8abff3a492998abff72e5c"}
Jul 14 01:04:27.969324 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:04:27.969288    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-jfg2f" podUID="f0ebac4e-8f5b-4820-b018-8386d03067f1"
Jul 14 01:04:29.323104 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:29.323013    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:29.912873 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:29.912845    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jjkx4"
Jul 14 01:04:29.947308 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:29.947279    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj" event={"ID":"32a8e14b-c44c-48c4-90af-656a3555d54e","Type":"ContainerStarted","Data":"39f7afb77f9d9d18d0d29baf16e88c29247df44bacc0c487b454bb2401eeab27"}
Jul 14 01:04:29.947468 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:29.947327    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj" event={"ID":"32a8e14b-c44c-48c4-90af-656a3555d54e","Type":"ContainerStarted","Data":"2b4373466075318454cc2e3f9dd49998e7d3334b68ace2f294197b59991aaf9a"}
Jul 14 01:04:29.961978 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:29.961851    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-k24bj" podStartSLOduration=1.836877298 podStartE2EDuration="3.961829127s" totalImagesPullingTime="2.124951829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:04:29.961197448 +0000 UTC m=+299.525606503" watchObservedRunningTime="2026-07-14 01:04:29.961829127 +0000 UTC m=+299.526238191"
Jul 14 01:04:30.869872 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:30.869845    2567 kubelet.go:1648] "Image garbage collection succeeded"
Jul 14 01:04:31.567153 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.567110    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:31.826600 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.826520    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4fcbl"]
Jul 14 01:04:31.829540 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.829517    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:31.831306 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.831287    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 14 01:04:31.831437 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.831360    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s7dcf\""
Jul 14 01:04:31.831525 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.831510    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 14 01:04:31.831584 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.831529    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 14 01:04:31.928994 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.928937    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/047eff54-d982-4e1f-a152-46ffd29c8db2-node-exporter-textfile\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:31.929379 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.929051    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/047eff54-d982-4e1f-a152-46ffd29c8db2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:31.929379 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.929075    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/047eff54-d982-4e1f-a152-46ffd29c8db2-node-exporter-accelerators-collector-config\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:31.929379 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.929108    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8c79\" (UniqueName: \"kubernetes.io/projected/047eff54-d982-4e1f-a152-46ffd29c8db2-kube-api-access-l8c79\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:31.929379 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.929151    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/047eff54-d982-4e1f-a152-46ffd29c8db2-root\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:31.929379 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.929180    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/047eff54-d982-4e1f-a152-46ffd29c8db2-node-exporter-tls\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:31.929379 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.929202    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/047eff54-d982-4e1f-a152-46ffd29c8db2-sys\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:31.929379 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.929221    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/047eff54-d982-4e1f-a152-46ffd29c8db2-node-exporter-wtmp\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:31.929379 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.929258    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/047eff54-d982-4e1f-a152-46ffd29c8db2-metrics-client-ca\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:31.930663 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:31.930639    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:32.030130 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.030085    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/047eff54-d982-4e1f-a152-46ffd29c8db2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.030130 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.030129    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/047eff54-d982-4e1f-a152-46ffd29c8db2-node-exporter-accelerators-collector-config\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.030418 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.030154    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8c79\" (UniqueName: \"kubernetes.io/projected/047eff54-d982-4e1f-a152-46ffd29c8db2-kube-api-access-l8c79\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.030418 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.030283    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/047eff54-d982-4e1f-a152-46ffd29c8db2-root\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.030418 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.030316    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/047eff54-d982-4e1f-a152-46ffd29c8db2-node-exporter-tls\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.030418 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.030344    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/047eff54-d982-4e1f-a152-46ffd29c8db2-sys\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.030418 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.030367    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/047eff54-d982-4e1f-a152-46ffd29c8db2-node-exporter-wtmp\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.030418 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.030401    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/047eff54-d982-4e1f-a152-46ffd29c8db2-metrics-client-ca\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.030418 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.030409    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/047eff54-d982-4e1f-a152-46ffd29c8db2-root\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.030779 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.030445    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/047eff54-d982-4e1f-a152-46ffd29c8db2-node-exporter-textfile\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.030779 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.030515    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/047eff54-d982-4e1f-a152-46ffd29c8db2-sys\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.030779 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.030527    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/047eff54-d982-4e1f-a152-46ffd29c8db2-node-exporter-wtmp\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.030779 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.030769    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/047eff54-d982-4e1f-a152-46ffd29c8db2-node-exporter-textfile\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.031013 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.030769    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/047eff54-d982-4e1f-a152-46ffd29c8db2-node-exporter-accelerators-collector-config\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.031013 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.030993    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/047eff54-d982-4e1f-a152-46ffd29c8db2-metrics-client-ca\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.032991 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.032949    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/047eff54-d982-4e1f-a152-46ffd29c8db2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.033056 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.033043    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/047eff54-d982-4e1f-a152-46ffd29c8db2-node-exporter-tls\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.037664 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.037644    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8c79\" (UniqueName: \"kubernetes.io/projected/047eff54-d982-4e1f-a152-46ffd29c8db2-kube-api-access-l8c79\") pod \"node-exporter-4fcbl\" (UID: \"047eff54-d982-4e1f-a152-46ffd29c8db2\") " pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.138768 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.138726    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4fcbl"
Jul 14 01:04:32.149837 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:04:32.149810    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047eff54_d982_4e1f_a152_46ffd29c8db2.slice/crio-25d67cc8bd6a7e8ac6efca57af31276da87ca08b3569b9cc1fa0a8aacf1404fc WatchSource:0}: Error finding container 25d67cc8bd6a7e8ac6efca57af31276da87ca08b3569b9cc1fa0a8aacf1404fc: Status 404 returned error can't find the container with id 25d67cc8bd6a7e8ac6efca57af31276da87ca08b3569b9cc1fa0a8aacf1404fc
Jul 14 01:04:32.151909 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.151887    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 01:04:32.170655 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.170622    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:32.270237 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.270203    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:32.495404 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.495310    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:32.598352 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.598324    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:32.800319 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.800238    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:32.900116 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.900080    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:32.958965 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:32.958907    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4fcbl" event={"ID":"047eff54-d982-4e1f-a152-46ffd29c8db2","Type":"ContainerStarted","Data":"25d67cc8bd6a7e8ac6efca57af31276da87ca08b3569b9cc1fa0a8aacf1404fc"}
Jul 14 01:04:33.024238 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:33.024169    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:33.875565 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:33.875532    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:33.963793 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:33.963748    2567 generic.go:356] "Generic (PLEG): container finished" podID="047eff54-d982-4e1f-a152-46ffd29c8db2" containerID="e6a9719a538bd0a518d310710692a1a13c7311349ab80a7bf7df8c23895ccb43" exitCode=0
Jul 14 01:04:33.964262 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:33.963837    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4fcbl" event={"ID":"047eff54-d982-4e1f-a152-46ffd29c8db2","Type":"ContainerDied","Data":"e6a9719a538bd0a518d310710692a1a13c7311349ab80a7bf7df8c23895ccb43"}
Jul 14 01:04:34.974627 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:34.974591    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4fcbl" event={"ID":"047eff54-d982-4e1f-a152-46ffd29c8db2","Type":"ContainerStarted","Data":"177ccf1a7c39fb33a963a6a5308babb178bf5b2e5153d543020b8b3240517b37"}
Jul 14 01:04:34.975025 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:34.974650    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4fcbl" event={"ID":"047eff54-d982-4e1f-a152-46ffd29c8db2","Type":"ContainerStarted","Data":"70ee61c217b37cb58c8603164c21a9a3f27be7a7e40cf1f9b72a6afff8af325b"}
Jul 14 01:04:34.991840 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:34.991776    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4fcbl" podStartSLOduration=2.914301009 podStartE2EDuration="3.991759329s" totalImagesPullingTime="1.004693299s" totalInitContainerRuntime="72.765021ms" isStatefulPod=true podCreationTimestamp="2026-07-14 01:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:04:34.990586279 +0000 UTC m=+304.554995356" watchObservedRunningTime="2026-07-14 01:04:34.991759329 +0000 UTC m=+304.556168383"
Jul 14 01:04:35.982065 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:35.982029    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d59cbd55f-nfdch"]
Jul 14 01:04:35.986552 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:35.986514    2567 patch_prober.go:28] interesting pod/image-registry-6d59cbd55f-nfdch container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 14 01:04:35.986712 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:35.986576    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch" podUID="4f83ab17-7261-408c-aaad-f38a36de4213" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 01:04:36.114595 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:36.114560    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:36.591677 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:36.591645    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6"]
Jul 14 01:04:36.598741 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:36.598721    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6"
Jul 14 01:04:36.600699 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:36.600673    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 14 01:04:36.600833 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:36.600745    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mqjfr\""
Jul 14 01:04:36.604570 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:36.604542    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6"]
Jul 14 01:04:36.608529 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:36.608502    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:36.657263 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:36.657226    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:36.667859 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:36.667832    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4vf2\" (UniqueName: \"kubernetes.io/projected/a5b6f961-4322-40e4-a5b1-12efb6f8e785-kube-api-access-k4vf2\") pod \"monitoring-plugin-5f68df7f58-t4nt6\" (UID: \"a5b6f961-4322-40e4-a5b1-12efb6f8e785\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6"
Jul 14 01:04:36.667988 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:36.667865    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5b6f961-4322-40e4-a5b1-12efb6f8e785-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-t4nt6\" (UID: \"a5b6f961-4322-40e4-a5b1-12efb6f8e785\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6"
Jul 14 01:04:36.769173 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:36.769122    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4vf2\" (UniqueName: \"kubernetes.io/projected/a5b6f961-4322-40e4-a5b1-12efb6f8e785-kube-api-access-k4vf2\") pod \"monitoring-plugin-5f68df7f58-t4nt6\" (UID: \"a5b6f961-4322-40e4-a5b1-12efb6f8e785\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6"
Jul 14 01:04:36.769173 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:36.769176    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5b6f961-4322-40e4-a5b1-12efb6f8e785-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-t4nt6\" (UID: \"a5b6f961-4322-40e4-a5b1-12efb6f8e785\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6"
Jul 14 01:04:36.769368 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:04:36.769304    2567 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 14 01:04:36.769368 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:04:36.769357    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5b6f961-4322-40e4-a5b1-12efb6f8e785-monitoring-plugin-cert podName:a5b6f961-4322-40e4-a5b1-12efb6f8e785 nodeName:}" failed. No retries permitted until 2026-07-14 01:04:37.269342882 +0000 UTC m=+306.833751916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a5b6f961-4322-40e4-a5b1-12efb6f8e785-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-t4nt6" (UID: "a5b6f961-4322-40e4-a5b1-12efb6f8e785") : secret "monitoring-plugin-cert" not found
Jul 14 01:04:36.777377 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:36.777351    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4vf2\" (UniqueName: \"kubernetes.io/projected/a5b6f961-4322-40e4-a5b1-12efb6f8e785-kube-api-access-k4vf2\") pod \"monitoring-plugin-5f68df7f58-t4nt6\" (UID: \"a5b6f961-4322-40e4-a5b1-12efb6f8e785\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6"
Jul 14 01:04:37.147122 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:37.147080    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:37.272520 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:37.272481    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5b6f961-4322-40e4-a5b1-12efb6f8e785-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-t4nt6\" (UID: \"a5b6f961-4322-40e4-a5b1-12efb6f8e785\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6"
Jul 14 01:04:37.275082 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:37.275051    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5b6f961-4322-40e4-a5b1-12efb6f8e785-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-t4nt6\" (UID: \"a5b6f961-4322-40e4-a5b1-12efb6f8e785\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6"
Jul 14 01:04:37.510478 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:37.510386    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6"
Jul 14 01:04:37.641172 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:37.641126    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6"]
Jul 14 01:04:37.644921 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:04:37.644891    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b6f961_4322_40e4_a5b1_12efb6f8e785.slice/crio-a01f0100d9c5d09dcbe23ba9a6281e113354400d56436a6378f4d9d87ae519e5 WatchSource:0}: Error finding container a01f0100d9c5d09dcbe23ba9a6281e113354400d56436a6378f4d9d87ae519e5: Status 404 returned error can't find the container with id a01f0100d9c5d09dcbe23ba9a6281e113354400d56436a6378f4d9d87ae519e5
Jul 14 01:04:37.986620 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:37.986582    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6" event={"ID":"a5b6f961-4322-40e4-a5b1-12efb6f8e785","Type":"ContainerStarted","Data":"a01f0100d9c5d09dcbe23ba9a6281e113354400d56436a6378f4d9d87ae519e5"}
Jul 14 01:04:38.033805 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:38.033760    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:39.051479 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:39.051446    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:39.153949 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:39.153905    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:39.969494 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:39.969461    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 01:04:39.996234 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:39.996202    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6" event={"ID":"a5b6f961-4322-40e4-a5b1-12efb6f8e785","Type":"ContainerStarted","Data":"24864205238b0c54188de6d493a6b165bcb117eb864de721105c9864838e4bb8"}
Jul 14 01:04:39.996392 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:39.996369    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6"
Jul 14 01:04:40.001249 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:40.001221    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6"
Jul 14 01:04:40.009867 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:40.009823    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-t4nt6" podStartSLOduration=2.478803414 podStartE2EDuration="4.009809716s" totalImagesPullingTime="1.531006302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:04:40.008995224 +0000 UTC m=+309.573404278" watchObservedRunningTime="2026-07-14 01:04:40.009809716 +0000 UTC m=+309.574218771"
Jul 14 01:04:43.718546 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:43.718513    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:43.818922 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:43.818881    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:45.987078 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:45.987051    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:04:47.177488 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:47.177448    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:48.822881 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:48.822843    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:48.899836 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:48.899792    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:50.583809 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.583762    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:50.631196 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.631163    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:50.781061 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.781029    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b6464df6-lk5gq"]
Jul 14 01:04:50.801607 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.801570    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b6464df6-lk5gq"]
Jul 14 01:04:50.801762 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.801687    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.804675 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.804646    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 14 01:04:50.805065 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.805040    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 14 01:04:50.805065 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.805054    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 14 01:04:50.805322 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.805138    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 14 01:04:50.805322 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.805184    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 14 01:04:50.805559 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.805541    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 14 01:04:50.805627 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.805594    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zqnn7\""
Jul 14 01:04:50.805740 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.805724    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 14 01:04:50.810773 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.810750    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 14 01:04:50.893088 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.893049    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-service-ca\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.893292 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.893101    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-config\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.893292 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.893129    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-serving-cert\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.893292 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.893212    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p79rg\" (UniqueName: \"kubernetes.io/projected/2544741a-cd89-42d7-a7ce-1f628d6aafff-kube-api-access-p79rg\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.893292 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.893247    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-oauth-serving-cert\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.893292 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.893280    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-trusted-ca-bundle\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.893526 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.893323    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-oauth-config\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.993852 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.993814    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p79rg\" (UniqueName: \"kubernetes.io/projected/2544741a-cd89-42d7-a7ce-1f628d6aafff-kube-api-access-p79rg\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.993852 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.993856    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-oauth-serving-cert\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.994135 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.993885    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-trusted-ca-bundle\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.994135 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.994007    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-oauth-config\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.994135 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.994056    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-service-ca\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.994135 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.994088    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-config\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.994135 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.994112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-serving-cert\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.994683 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.994662    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-oauth-serving-cert\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.994795 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.994761    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-service-ca\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.994795 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.994785    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-config\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.994879 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.994764    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-trusted-ca-bundle\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.996562 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.996541    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-serving-cert\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:50.996663 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:50.996646    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-oauth-config\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:51.001416 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:51.001385    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p79rg\" (UniqueName: \"kubernetes.io/projected/2544741a-cd89-42d7-a7ce-1f628d6aafff-kube-api-access-p79rg\") pod \"console-7b6464df6-lk5gq\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") " pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:51.112310 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:51.112270    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:04:51.246219 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:51.246183    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b6464df6-lk5gq"]
Jul 14 01:04:51.249449 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:04:51.249421    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2544741a_cd89_42d7_a7ce_1f628d6aafff.slice/crio-9a7021d688737e0fbb4cc73842a30d3a8d4dc548392e255a2e11a304a17ab786 WatchSource:0}: Error finding container 9a7021d688737e0fbb4cc73842a30d3a8d4dc548392e255a2e11a304a17ab786: Status 404 returned error can't find the container with id 9a7021d688737e0fbb4cc73842a30d3a8d4dc548392e255a2e11a304a17ab786
Jul 14 01:04:52.037750 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:52.037720    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6464df6-lk5gq" event={"ID":"2544741a-cd89-42d7-a7ce-1f628d6aafff","Type":"ContainerStarted","Data":"9a7021d688737e0fbb4cc73842a30d3a8d4dc548392e255a2e11a304a17ab786"}
Jul 14 01:04:53.589217 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:53.589176    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:53.690223 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:53.690174    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:55.051055 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:55.051013    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6464df6-lk5gq" event={"ID":"2544741a-cd89-42d7-a7ce-1f628d6aafff","Type":"ContainerStarted","Data":"4330f3178ec4ff90180b867ac2735cfad3ae3a5f6a2eb903988dbed2439f3718"}
Jul 14 01:04:55.067805 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:55.067729    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b6464df6-lk5gq" podStartSLOduration=1.787520329 podStartE2EDuration="5.067709286s" totalImagesPullingTime="3.280188957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:04:55.067367224 +0000 UTC m=+324.631776279" watchObservedRunningTime="2026-07-14 01:04:55.067709286 +0000 UTC m=+324.632118342"
Jul 14 01:04:57.883695 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:57.883659    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:57.932480 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:57.932423    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:58.631757 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:58.631723    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:04:58.731088 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:04:58.731055    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:00.793289 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:00.793259    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:00.933727 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:00.933694    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:01.002566 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.002521    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch" podUID="4f83ab17-7261-408c-aaad-f38a36de4213" containerName="registry" containerID="cri-o://49463c6a23016b02e7893e140c65dca2e70ff3b1d54a3ac45baa4fe321c2c2fe" gracePeriod=30
Jul 14 01:05:01.113519 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.113424    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:05:01.113659 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.113536    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:05:01.118496 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.118469    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:05:01.253777 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.253753    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:05:01.384441 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.384406    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f83ab17-7261-408c-aaad-f38a36de4213-installation-pull-secrets\") pod \"4f83ab17-7261-408c-aaad-f38a36de4213\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") "
Jul 14 01:05:01.384637 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.384464    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-bound-sa-token\") pod \"4f83ab17-7261-408c-aaad-f38a36de4213\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") "
Jul 14 01:05:01.384637 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.384488    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f83ab17-7261-408c-aaad-f38a36de4213-image-registry-private-configuration\") pod \"4f83ab17-7261-408c-aaad-f38a36de4213\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") "
Jul 14 01:05:01.384637 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.384506    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f83ab17-7261-408c-aaad-f38a36de4213-registry-certificates\") pod \"4f83ab17-7261-408c-aaad-f38a36de4213\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") "
Jul 14 01:05:01.384637 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.384524    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls\") pod \"4f83ab17-7261-408c-aaad-f38a36de4213\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") "
Jul 14 01:05:01.384637 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.384557    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f83ab17-7261-408c-aaad-f38a36de4213-trusted-ca\") pod \"4f83ab17-7261-408c-aaad-f38a36de4213\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") "
Jul 14 01:05:01.384637 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.384584    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f83ab17-7261-408c-aaad-f38a36de4213-ca-trust-extracted\") pod \"4f83ab17-7261-408c-aaad-f38a36de4213\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") "
Jul 14 01:05:01.384929 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.384721    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdc99\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-kube-api-access-pdc99\") pod \"4f83ab17-7261-408c-aaad-f38a36de4213\" (UID: \"4f83ab17-7261-408c-aaad-f38a36de4213\") "
Jul 14 01:05:01.385172 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.385134    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f83ab17-7261-408c-aaad-f38a36de4213-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4f83ab17-7261-408c-aaad-f38a36de4213" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:05:01.385840 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.385806    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f83ab17-7261-408c-aaad-f38a36de4213-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4f83ab17-7261-408c-aaad-f38a36de4213" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:05:01.387232 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.387194    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f83ab17-7261-408c-aaad-f38a36de4213-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4f83ab17-7261-408c-aaad-f38a36de4213" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:05:01.387367 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.387341    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4f83ab17-7261-408c-aaad-f38a36de4213" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:05:01.387518 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.387495    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-kube-api-access-pdc99" (OuterVolumeSpecName: "kube-api-access-pdc99") pod "4f83ab17-7261-408c-aaad-f38a36de4213" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213"). InnerVolumeSpecName "kube-api-access-pdc99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:05:01.387602 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.387582    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4f83ab17-7261-408c-aaad-f38a36de4213" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:05:01.387652 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.387619    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f83ab17-7261-408c-aaad-f38a36de4213-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4f83ab17-7261-408c-aaad-f38a36de4213" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:05:01.393754 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.393726    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f83ab17-7261-408c-aaad-f38a36de4213-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4f83ab17-7261-408c-aaad-f38a36de4213" (UID: "4f83ab17-7261-408c-aaad-f38a36de4213"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:05:01.486180 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.486143    2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-registry-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:05:01.486180 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.486172    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f83ab17-7261-408c-aaad-f38a36de4213-trusted-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:05:01.486180 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.486181    2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4f83ab17-7261-408c-aaad-f38a36de4213-ca-trust-extracted\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:05:01.486180 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.486192    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdc99\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-kube-api-access-pdc99\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:05:01.486434 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.486201    2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4f83ab17-7261-408c-aaad-f38a36de4213-installation-pull-secrets\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:05:01.486434 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.486210    2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f83ab17-7261-408c-aaad-f38a36de4213-bound-sa-token\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:05:01.486434 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.486219    2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4f83ab17-7261-408c-aaad-f38a36de4213-image-registry-private-configuration\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:05:01.486434 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.486227    2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4f83ab17-7261-408c-aaad-f38a36de4213-registry-certificates\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:05:01.845310 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:01.845207    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:02.079088 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:02.079049    2567 generic.go:356] "Generic (PLEG): container finished" podID="4f83ab17-7261-408c-aaad-f38a36de4213" containerID="49463c6a23016b02e7893e140c65dca2e70ff3b1d54a3ac45baa4fe321c2c2fe" exitCode=0
Jul 14 01:05:02.079252 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:02.079090    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch"
Jul 14 01:05:02.079252 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:02.079140    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch" event={"ID":"4f83ab17-7261-408c-aaad-f38a36de4213","Type":"ContainerDied","Data":"49463c6a23016b02e7893e140c65dca2e70ff3b1d54a3ac45baa4fe321c2c2fe"}
Jul 14 01:05:02.079252 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:02.079182    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d59cbd55f-nfdch" event={"ID":"4f83ab17-7261-408c-aaad-f38a36de4213","Type":"ContainerDied","Data":"cfb0ec41f960e91b3e6a1ff314420a7f7982b1d5963d6acbf70cb7d96cc4b8e1"}
Jul 14 01:05:02.079252 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:02.079216    2567 scope.go:117] "RemoveContainer" containerID="49463c6a23016b02e7893e140c65dca2e70ff3b1d54a3ac45baa4fe321c2c2fe"
Jul 14 01:05:02.083736 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:02.083697    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:05:02.088876 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:02.088861    2567 scope.go:117] "RemoveContainer" containerID="49463c6a23016b02e7893e140c65dca2e70ff3b1d54a3ac45baa4fe321c2c2fe"
Jul 14 01:05:02.089190 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:05:02.089172    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49463c6a23016b02e7893e140c65dca2e70ff3b1d54a3ac45baa4fe321c2c2fe\": container with ID starting with 49463c6a23016b02e7893e140c65dca2e70ff3b1d54a3ac45baa4fe321c2c2fe not found: ID does not exist" containerID="49463c6a23016b02e7893e140c65dca2e70ff3b1d54a3ac45baa4fe321c2c2fe"
Jul 14 01:05:02.089237 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:02.089198    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49463c6a23016b02e7893e140c65dca2e70ff3b1d54a3ac45baa4fe321c2c2fe"} err="failed to get container status \"49463c6a23016b02e7893e140c65dca2e70ff3b1d54a3ac45baa4fe321c2c2fe\": rpc error: code = NotFound desc = could not find container \"49463c6a23016b02e7893e140c65dca2e70ff3b1d54a3ac45baa4fe321c2c2fe\": container with ID starting with 49463c6a23016b02e7893e140c65dca2e70ff3b1d54a3ac45baa4fe321c2c2fe not found: ID does not exist"
Jul 14 01:05:02.114660 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:02.114561    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d59cbd55f-nfdch"]
Jul 14 01:05:02.119255 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:02.119217    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d59cbd55f-nfdch"]
Jul 14 01:05:02.224093 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:02.224061    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:02.335136 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:02.335097    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:02.384605 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:02.384577    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:02.948238 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:02.948204    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:02.973697 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:02.973662    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f83ab17-7261-408c-aaad-f38a36de4213" path="/var/lib/kubelet/pods/4f83ab17-7261-408c-aaad-f38a36de4213/volumes"
Jul 14 01:05:03.057896 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:03.057841    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:13.632147 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:13.632112    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:13.688993 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:13.688946    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:16.649130 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:16.649098    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:24.008597 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:24.008516    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:25.660387 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:25.660326    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:25.724358 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:25.724323    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:44.922242 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:44.922199    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 01:05:44.924644 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:44.924624    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0ebac4e-8f5b-4820-b018-8386d03067f1-metrics-certs\") pod \"network-metrics-daemon-jfg2f\" (UID: \"f0ebac4e-8f5b-4820-b018-8386d03067f1\") " pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 01:05:45.072020 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:45.071985    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6f2s8\""
Jul 14 01:05:45.080442 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:45.080412    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfg2f"
Jul 14 01:05:45.222599 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:45.222567    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jfg2f"]
Jul 14 01:05:45.225993 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:05:45.225938    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ebac4e_8f5b_4820_b018_8386d03067f1.slice/crio-131221c0dd905ed0411d4b3cf2cf876d65d0d9acf0c24a6555c526a72c6eec1b WatchSource:0}: Error finding container 131221c0dd905ed0411d4b3cf2cf876d65d0d9acf0c24a6555c526a72c6eec1b: Status 404 returned error can't find the container with id 131221c0dd905ed0411d4b3cf2cf876d65d0d9acf0c24a6555c526a72c6eec1b
Jul 14 01:05:45.232311 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:45.232279    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jfg2f" event={"ID":"f0ebac4e-8f5b-4820-b018-8386d03067f1","Type":"ContainerStarted","Data":"131221c0dd905ed0411d4b3cf2cf876d65d0d9acf0c24a6555c526a72c6eec1b"}
Jul 14 01:05:47.243059 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:47.243025    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jfg2f" event={"ID":"f0ebac4e-8f5b-4820-b018-8386d03067f1","Type":"ContainerStarted","Data":"f90a595e23575f8950696f3a01aa6292abaff32861a3a188efaccde0ef72d191"}
Jul 14 01:05:47.243426 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:47.243075    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jfg2f" event={"ID":"f0ebac4e-8f5b-4820-b018-8386d03067f1","Type":"ContainerStarted","Data":"1786f3c5f6ca3e4f9fedc7982c9e33406b0c327b57a531b17db11f4524626869"}
Jul 14 01:05:47.259184 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:47.259125    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jfg2f" podStartSLOduration=375.271440555 podStartE2EDuration="6m16.25910799s" totalImagesPullingTime="987.667435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:05:47.257661777 +0000 UTC m=+376.822070833" watchObservedRunningTime="2026-07-14 01:05:47.25910799 +0000 UTC m=+376.823517042"
Jul 14 01:05:47.360433 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:47.360403    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:47.449147 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:47.449115    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:47.930816 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:47.930788    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:47.983240 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:47.983209    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:55.694599 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.694558    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-867df57d85-mt7qf"]
Jul 14 01:05:55.695198 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.694977    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f83ab17-7261-408c-aaad-f38a36de4213" containerName="registry"
Jul 14 01:05:55.695198 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.694994    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f83ab17-7261-408c-aaad-f38a36de4213" containerName="registry"
Jul 14 01:05:55.695198 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.695054    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f83ab17-7261-408c-aaad-f38a36de4213" containerName="registry"
Jul 14 01:05:55.698341 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.698310    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.708559 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.708530    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-867df57d85-mt7qf"]
Jul 14 01:05:55.743769 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.743732    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:55.807676 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.807639    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-service-ca\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.807676 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.807676    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-oauth-config\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.807890 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.807698    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-trusted-ca-bundle\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.807890 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.807750    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-config\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.807890 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.807803    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-oauth-serving-cert\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.807890 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.807855    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-serving-cert\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.807890 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.807877    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6m86\" (UniqueName: \"kubernetes.io/projected/e7c58e8d-5d60-401b-aabc-b83fbe67712c-kube-api-access-p6m86\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.842569 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.842534    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:55.909202 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.909163    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-serving-cert\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.909202 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.909209    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6m86\" (UniqueName: \"kubernetes.io/projected/e7c58e8d-5d60-401b-aabc-b83fbe67712c-kube-api-access-p6m86\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.909459 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.909239    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-service-ca\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.909459 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.909264    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-oauth-config\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.909459 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.909292    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-trusted-ca-bundle\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.909459 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.909315    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-config\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.909459 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.909352    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-oauth-serving-cert\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.910163 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.910115    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-config\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.910288 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.910231    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-service-ca\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.910288 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.910269    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-trusted-ca-bundle\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.910415 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.910394    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-oauth-serving-cert\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.911788 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.911765    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-serving-cert\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.911938 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.911920    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-oauth-config\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.916919 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.916885    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6m86\" (UniqueName: \"kubernetes.io/projected/e7c58e8d-5d60-401b-aabc-b83fbe67712c-kube-api-access-p6m86\") pod \"console-867df57d85-mt7qf\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") " pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:55.943143 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:55.943104    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:56.007545 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:56.007445    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:05:56.142302 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:56.142265    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-867df57d85-mt7qf"]
Jul 14 01:05:56.146830 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:05:56.146798    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c58e8d_5d60_401b_aabc_b83fbe67712c.slice/crio-e5e379a5eb78a2376b2ad5dc16600ba46bec3c9846a6b8fc856e88951ad7f857 WatchSource:0}: Error finding container e5e379a5eb78a2376b2ad5dc16600ba46bec3c9846a6b8fc856e88951ad7f857: Status 404 returned error can't find the container with id e5e379a5eb78a2376b2ad5dc16600ba46bec3c9846a6b8fc856e88951ad7f857
Jul 14 01:05:56.275896 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:56.275792    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-867df57d85-mt7qf" event={"ID":"e7c58e8d-5d60-401b-aabc-b83fbe67712c","Type":"ContainerStarted","Data":"92d03d7c09157ff24229d6520491c21f65880d13d8213a8a562a8c29ff33a64f"}
Jul 14 01:05:56.275896 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:56.275859    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-867df57d85-mt7qf" event={"ID":"e7c58e8d-5d60-401b-aabc-b83fbe67712c","Type":"ContainerStarted","Data":"e5e379a5eb78a2376b2ad5dc16600ba46bec3c9846a6b8fc856e88951ad7f857"}
Jul 14 01:05:56.292911 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:56.292849    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-867df57d85-mt7qf" podStartSLOduration=1.292826217 podStartE2EDuration="1.292826217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:05:56.290976585 +0000 UTC m=+385.855385634" watchObservedRunningTime="2026-07-14 01:05:56.292826217 +0000 UTC m=+385.857235275"
Jul 14 01:05:56.515204 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:56.515164    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:56.656234 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:56.656197    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:58.427243 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:58.427208    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:59.790424 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:59.790391    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:05:59.849648 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:05:59.849615    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:06:06.008350 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:06.008304    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:06:06.008761 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:06.008399    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:06:06.013072 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:06.013048    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:06:06.317792 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:06.317709    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:06:06.358228 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:06.358196    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b6464df6-lk5gq"]
Jul 14 01:06:06.469133 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:06.469086    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:06:06.517453 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:06.517406    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:06:06.569913 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:06.569832    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:06:13.640319 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:13.640266    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:06:13.739752 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:13.739714    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:06:31.385675 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.385623    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b6464df6-lk5gq" podUID="2544741a-cd89-42d7-a7ce-1f628d6aafff" containerName="console" containerID="cri-o://4330f3178ec4ff90180b867ac2735cfad3ae3a5f6a2eb903988dbed2439f3718" gracePeriod=15
Jul 14 01:06:31.656274 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.656247    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:06:31.790469 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.790434    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-service-ca\") pod \"2544741a-cd89-42d7-a7ce-1f628d6aafff\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") "
Jul 14 01:06:31.790670 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.790485    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-serving-cert\") pod \"2544741a-cd89-42d7-a7ce-1f628d6aafff\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") "
Jul 14 01:06:31.790670 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.790519    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p79rg\" (UniqueName: \"kubernetes.io/projected/2544741a-cd89-42d7-a7ce-1f628d6aafff-kube-api-access-p79rg\") pod \"2544741a-cd89-42d7-a7ce-1f628d6aafff\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") "
Jul 14 01:06:31.790670 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.790542    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-config\") pod \"2544741a-cd89-42d7-a7ce-1f628d6aafff\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") "
Jul 14 01:06:31.790670 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.790561    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-oauth-config\") pod \"2544741a-cd89-42d7-a7ce-1f628d6aafff\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") "
Jul 14 01:06:31.790670 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.790581    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-oauth-serving-cert\") pod \"2544741a-cd89-42d7-a7ce-1f628d6aafff\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") "
Jul 14 01:06:31.790670 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.790669    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-trusted-ca-bundle\") pod \"2544741a-cd89-42d7-a7ce-1f628d6aafff\" (UID: \"2544741a-cd89-42d7-a7ce-1f628d6aafff\") "
Jul 14 01:06:31.791008 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.790815    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-service-ca" (OuterVolumeSpecName: "service-ca") pod "2544741a-cd89-42d7-a7ce-1f628d6aafff" (UID: "2544741a-cd89-42d7-a7ce-1f628d6aafff"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:06:31.791084 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.791034    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2544741a-cd89-42d7-a7ce-1f628d6aafff" (UID: "2544741a-cd89-42d7-a7ce-1f628d6aafff"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:06:31.791142 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.791089    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-config" (OuterVolumeSpecName: "console-config") pod "2544741a-cd89-42d7-a7ce-1f628d6aafff" (UID: "2544741a-cd89-42d7-a7ce-1f628d6aafff"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:06:31.791206 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.791184    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2544741a-cd89-42d7-a7ce-1f628d6aafff" (UID: "2544741a-cd89-42d7-a7ce-1f628d6aafff"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:06:31.793053 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.793028    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2544741a-cd89-42d7-a7ce-1f628d6aafff-kube-api-access-p79rg" (OuterVolumeSpecName: "kube-api-access-p79rg") pod "2544741a-cd89-42d7-a7ce-1f628d6aafff" (UID: "2544741a-cd89-42d7-a7ce-1f628d6aafff"). InnerVolumeSpecName "kube-api-access-p79rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:06:31.793137 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.793092    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2544741a-cd89-42d7-a7ce-1f628d6aafff" (UID: "2544741a-cd89-42d7-a7ce-1f628d6aafff"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:06:31.793137 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.793123    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2544741a-cd89-42d7-a7ce-1f628d6aafff" (UID: "2544741a-cd89-42d7-a7ce-1f628d6aafff"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:06:31.891566 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.891465    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-trusted-ca-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:06:31.891566 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.891501    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:06:31.891566 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.891514    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-serving-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:06:31.891566 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.891525    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p79rg\" (UniqueName: \"kubernetes.io/projected/2544741a-cd89-42d7-a7ce-1f628d6aafff-kube-api-access-p79rg\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:06:31.891566 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.891536    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:06:31.891566 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.891548    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2544741a-cd89-42d7-a7ce-1f628d6aafff-console-oauth-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:06:31.891566 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:31.891559    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2544741a-cd89-42d7-a7ce-1f628d6aafff-oauth-serving-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:06:32.409476 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:32.409438    2567 generic.go:356] "Generic (PLEG): container finished" podID="2544741a-cd89-42d7-a7ce-1f628d6aafff" containerID="4330f3178ec4ff90180b867ac2735cfad3ae3a5f6a2eb903988dbed2439f3718" exitCode=0
Jul 14 01:06:32.409476 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:32.409481    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6464df6-lk5gq"
Jul 14 01:06:32.409980 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:32.409517    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6464df6-lk5gq" event={"ID":"2544741a-cd89-42d7-a7ce-1f628d6aafff","Type":"ContainerDied","Data":"4330f3178ec4ff90180b867ac2735cfad3ae3a5f6a2eb903988dbed2439f3718"}
Jul 14 01:06:32.409980 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:32.409567    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6464df6-lk5gq" event={"ID":"2544741a-cd89-42d7-a7ce-1f628d6aafff","Type":"ContainerDied","Data":"9a7021d688737e0fbb4cc73842a30d3a8d4dc548392e255a2e11a304a17ab786"}
Jul 14 01:06:32.409980 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:32.409597    2567 scope.go:117] "RemoveContainer" containerID="4330f3178ec4ff90180b867ac2735cfad3ae3a5f6a2eb903988dbed2439f3718"
Jul 14 01:06:32.418107 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:32.418053    2567 scope.go:117] "RemoveContainer" containerID="4330f3178ec4ff90180b867ac2735cfad3ae3a5f6a2eb903988dbed2439f3718"
Jul 14 01:06:32.418351 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:06:32.418336    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4330f3178ec4ff90180b867ac2735cfad3ae3a5f6a2eb903988dbed2439f3718\": container with ID starting with 4330f3178ec4ff90180b867ac2735cfad3ae3a5f6a2eb903988dbed2439f3718 not found: ID does not exist" containerID="4330f3178ec4ff90180b867ac2735cfad3ae3a5f6a2eb903988dbed2439f3718"
Jul 14 01:06:32.418418 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:32.418358    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4330f3178ec4ff90180b867ac2735cfad3ae3a5f6a2eb903988dbed2439f3718"} err="failed to get container status \"4330f3178ec4ff90180b867ac2735cfad3ae3a5f6a2eb903988dbed2439f3718\": rpc error: code = NotFound desc = could not find container \"4330f3178ec4ff90180b867ac2735cfad3ae3a5f6a2eb903988dbed2439f3718\": container with ID starting with 4330f3178ec4ff90180b867ac2735cfad3ae3a5f6a2eb903988dbed2439f3718 not found: ID does not exist"
Jul 14 01:06:32.447330 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:32.447294    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b6464df6-lk5gq"]
Jul 14 01:06:32.452263 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:32.452237    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b6464df6-lk5gq"]
Jul 14 01:06:32.973448 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:32.973411    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2544741a-cd89-42d7-a7ce-1f628d6aafff" path="/var/lib/kubelet/pods/2544741a-cd89-42d7-a7ce-1f628d6aafff/volumes"
Jul 14 01:06:48.224331 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.224294    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:06:48.323763 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.323730    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:06:48.789502 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.789469    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"]
Jul 14 01:06:48.789761 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.789750    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2544741a-cd89-42d7-a7ce-1f628d6aafff" containerName="console"
Jul 14 01:06:48.789814 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.789763    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2544741a-cd89-42d7-a7ce-1f628d6aafff" containerName="console"
Jul 14 01:06:48.789850 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.789818    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2544741a-cd89-42d7-a7ce-1f628d6aafff" containerName="console"
Jul 14 01:06:48.793969 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.793931    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"
Jul 14 01:06:48.796242 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.796207    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hrggj\""
Jul 14 01:06:48.796242 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.796226    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 14 01:06:48.796638 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.796623    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 14 01:06:48.800258 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.800222    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"]
Jul 14 01:06:48.829644 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.829602    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a6caa0c-1898-4bc5-9980-a051940bf04f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55\" (UID: \"8a6caa0c-1898-4bc5-9980-a051940bf04f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"
Jul 14 01:06:48.829815 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.829649    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a6caa0c-1898-4bc5-9980-a051940bf04f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55\" (UID: \"8a6caa0c-1898-4bc5-9980-a051940bf04f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"
Jul 14 01:06:48.829815 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.829711    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-289kl\" (UniqueName: \"kubernetes.io/projected/8a6caa0c-1898-4bc5-9980-a051940bf04f-kube-api-access-289kl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55\" (UID: \"8a6caa0c-1898-4bc5-9980-a051940bf04f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"
Jul 14 01:06:48.930067 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.930030    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-289kl\" (UniqueName: \"kubernetes.io/projected/8a6caa0c-1898-4bc5-9980-a051940bf04f-kube-api-access-289kl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55\" (UID: \"8a6caa0c-1898-4bc5-9980-a051940bf04f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"
Jul 14 01:06:48.930193 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.930107    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a6caa0c-1898-4bc5-9980-a051940bf04f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55\" (UID: \"8a6caa0c-1898-4bc5-9980-a051940bf04f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"
Jul 14 01:06:48.930193 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.930140    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a6caa0c-1898-4bc5-9980-a051940bf04f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55\" (UID: \"8a6caa0c-1898-4bc5-9980-a051940bf04f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"
Jul 14 01:06:48.930502 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.930480    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a6caa0c-1898-4bc5-9980-a051940bf04f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55\" (UID: \"8a6caa0c-1898-4bc5-9980-a051940bf04f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"
Jul 14 01:06:48.930537 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.930514    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a6caa0c-1898-4bc5-9980-a051940bf04f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55\" (UID: \"8a6caa0c-1898-4bc5-9980-a051940bf04f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"
Jul 14 01:06:48.938360 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:48.938319    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-289kl\" (UniqueName: \"kubernetes.io/projected/8a6caa0c-1898-4bc5-9980-a051940bf04f-kube-api-access-289kl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55\" (UID: \"8a6caa0c-1898-4bc5-9980-a051940bf04f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"
Jul 14 01:06:49.105732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:49.105626    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"
Jul 14 01:06:49.240439 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:49.240414    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"]
Jul 14 01:06:49.242769 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:06:49.242745    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6caa0c_1898_4bc5_9980_a051940bf04f.slice/crio-2a66222fc9e2f4a2a770b74adc919d2defb9d9f18a250880d0f92dee4e8d4d77 WatchSource:0}: Error finding container 2a66222fc9e2f4a2a770b74adc919d2defb9d9f18a250880d0f92dee4e8d4d77: Status 404 returned error can't find the container with id 2a66222fc9e2f4a2a770b74adc919d2defb9d9f18a250880d0f92dee4e8d4d77
Jul 14 01:06:49.477445 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:49.477408    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55" event={"ID":"8a6caa0c-1898-4bc5-9980-a051940bf04f","Type":"ContainerStarted","Data":"2a66222fc9e2f4a2a770b74adc919d2defb9d9f18a250880d0f92dee4e8d4d77"}
Jul 14 01:06:53.506015 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:53.505932    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:06:53.557909 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:53.557875    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:06:57.508157 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:57.508118    2567 generic.go:356] "Generic (PLEG): container finished" podID="8a6caa0c-1898-4bc5-9980-a051940bf04f" containerID="ed32a71ec9976cce69239db96e276db18d7a6272cbd3eedeb7f91bf2a8f1ff36" exitCode=0
Jul 14 01:06:57.508549 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:06:57.508209    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55" event={"ID":"8a6caa0c-1898-4bc5-9980-a051940bf04f","Type":"ContainerDied","Data":"ed32a71ec9976cce69239db96e276db18d7a6272cbd3eedeb7f91bf2a8f1ff36"}
Jul 14 01:07:00.520391 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:00.520296    2567 generic.go:356] "Generic (PLEG): container finished" podID="8a6caa0c-1898-4bc5-9980-a051940bf04f" containerID="a928200027880ab16d14095bbf776dd1f775c9cad4bef5b296051251ea6172a7" exitCode=0
Jul 14 01:07:00.520846 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:00.520399    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55" event={"ID":"8a6caa0c-1898-4bc5-9980-a051940bf04f","Type":"ContainerDied","Data":"a928200027880ab16d14095bbf776dd1f775c9cad4bef5b296051251ea6172a7"}
Jul 14 01:07:04.377332 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:04.377290    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:10.562085 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:10.562046    2567 generic.go:356] "Generic (PLEG): container finished" podID="8a6caa0c-1898-4bc5-9980-a051940bf04f" containerID="bdd6213e2d9da0c1888f4c6dd592fadf2c3ce78b7349b745534b233dbc058f3e" exitCode=0
Jul 14 01:07:10.562463 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:10.562156    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55" event={"ID":"8a6caa0c-1898-4bc5-9980-a051940bf04f","Type":"ContainerDied","Data":"bdd6213e2d9da0c1888f4c6dd592fadf2c3ce78b7349b745534b233dbc058f3e"}
Jul 14 01:07:11.693757 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:11.693729    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"
Jul 14 01:07:11.817481 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:11.817390    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a6caa0c-1898-4bc5-9980-a051940bf04f-bundle\") pod \"8a6caa0c-1898-4bc5-9980-a051940bf04f\" (UID: \"8a6caa0c-1898-4bc5-9980-a051940bf04f\") "
Jul 14 01:07:11.817643 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:11.817497    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-289kl\" (UniqueName: \"kubernetes.io/projected/8a6caa0c-1898-4bc5-9980-a051940bf04f-kube-api-access-289kl\") pod \"8a6caa0c-1898-4bc5-9980-a051940bf04f\" (UID: \"8a6caa0c-1898-4bc5-9980-a051940bf04f\") "
Jul 14 01:07:11.817643 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:11.817524    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a6caa0c-1898-4bc5-9980-a051940bf04f-util\") pod \"8a6caa0c-1898-4bc5-9980-a051940bf04f\" (UID: \"8a6caa0c-1898-4bc5-9980-a051940bf04f\") "
Jul 14 01:07:11.818159 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:11.818137    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6caa0c-1898-4bc5-9980-a051940bf04f-bundle" (OuterVolumeSpecName: "bundle") pod "8a6caa0c-1898-4bc5-9980-a051940bf04f" (UID: "8a6caa0c-1898-4bc5-9980-a051940bf04f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:07:11.819834 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:11.819808    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6caa0c-1898-4bc5-9980-a051940bf04f-kube-api-access-289kl" (OuterVolumeSpecName: "kube-api-access-289kl") pod "8a6caa0c-1898-4bc5-9980-a051940bf04f" (UID: "8a6caa0c-1898-4bc5-9980-a051940bf04f"). InnerVolumeSpecName "kube-api-access-289kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:07:11.822293 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:11.822258    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6caa0c-1898-4bc5-9980-a051940bf04f-util" (OuterVolumeSpecName: "util") pod "8a6caa0c-1898-4bc5-9980-a051940bf04f" (UID: "8a6caa0c-1898-4bc5-9980-a051940bf04f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:07:11.918838 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:11.918797    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-289kl\" (UniqueName: \"kubernetes.io/projected/8a6caa0c-1898-4bc5-9980-a051940bf04f-kube-api-access-289kl\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:07:11.918838 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:11.918829    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a6caa0c-1898-4bc5-9980-a051940bf04f-util\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:07:11.918838 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:11.918842    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a6caa0c-1898-4bc5-9980-a051940bf04f-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:07:12.571184 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:12.571158    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55"
Jul 14 01:07:12.571346 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:12.571161    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dcb55" event={"ID":"8a6caa0c-1898-4bc5-9980-a051940bf04f","Type":"ContainerDied","Data":"2a66222fc9e2f4a2a770b74adc919d2defb9d9f18a250880d0f92dee4e8d4d77"}
Jul 14 01:07:12.571346 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:12.571274    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a66222fc9e2f4a2a770b74adc919d2defb9d9f18a250880d0f92dee4e8d4d77"
Jul 14 01:07:19.306438 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:19.306403    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:22.224989 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.224940    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7lw6w"]
Jul 14 01:07:22.225464 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.225326    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6caa0c-1898-4bc5-9980-a051940bf04f" containerName="util"
Jul 14 01:07:22.225464 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.225342    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6caa0c-1898-4bc5-9980-a051940bf04f" containerName="util"
Jul 14 01:07:22.225464 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.225365    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6caa0c-1898-4bc5-9980-a051940bf04f" containerName="pull"
Jul 14 01:07:22.225464 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.225372    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6caa0c-1898-4bc5-9980-a051940bf04f" containerName="pull"
Jul 14 01:07:22.225464 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.225387    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6caa0c-1898-4bc5-9980-a051940bf04f" containerName="extract"
Jul 14 01:07:22.225464 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.225395    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6caa0c-1898-4bc5-9980-a051940bf04f" containerName="extract"
Jul 14 01:07:22.225464 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.225457    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6caa0c-1898-4bc5-9980-a051940bf04f" containerName="extract"
Jul 14 01:07:22.227440 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.227420    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7lw6w"
Jul 14 01:07:22.230311 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.230284    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 14 01:07:22.230432 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.230405    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 14 01:07:22.230491 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.230405    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-b2tx8\""
Jul 14 01:07:22.257375 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.257342    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7lw6w"]
Jul 14 01:07:22.301299 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.301248    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1408ccdc-23f8-4374-a582-13cc9bf26cf8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7lw6w\" (UID: \"1408ccdc-23f8-4374-a582-13cc9bf26cf8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7lw6w"
Jul 14 01:07:22.301299 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.301297    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjmjh\" (UniqueName: \"kubernetes.io/projected/1408ccdc-23f8-4374-a582-13cc9bf26cf8-kube-api-access-cjmjh\") pod \"cert-manager-operator-controller-manager-55c444bf86-7lw6w\" (UID: \"1408ccdc-23f8-4374-a582-13cc9bf26cf8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7lw6w"
Jul 14 01:07:22.402775 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.402719    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1408ccdc-23f8-4374-a582-13cc9bf26cf8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7lw6w\" (UID: \"1408ccdc-23f8-4374-a582-13cc9bf26cf8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7lw6w"
Jul 14 01:07:22.402775 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.402773    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjmjh\" (UniqueName: \"kubernetes.io/projected/1408ccdc-23f8-4374-a582-13cc9bf26cf8-kube-api-access-cjmjh\") pod \"cert-manager-operator-controller-manager-55c444bf86-7lw6w\" (UID: \"1408ccdc-23f8-4374-a582-13cc9bf26cf8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7lw6w"
Jul 14 01:07:22.403167 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.403149    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1408ccdc-23f8-4374-a582-13cc9bf26cf8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7lw6w\" (UID: \"1408ccdc-23f8-4374-a582-13cc9bf26cf8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7lw6w"
Jul 14 01:07:22.410500 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.410467    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjmjh\" (UniqueName: \"kubernetes.io/projected/1408ccdc-23f8-4374-a582-13cc9bf26cf8-kube-api-access-cjmjh\") pod \"cert-manager-operator-controller-manager-55c444bf86-7lw6w\" (UID: \"1408ccdc-23f8-4374-a582-13cc9bf26cf8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7lw6w"
Jul 14 01:07:22.536140 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.536038    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7lw6w"
Jul 14 01:07:22.675004 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:22.674950    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7lw6w"]
Jul 14 01:07:22.680294 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:07:22.680267    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1408ccdc_23f8_4374_a582_13cc9bf26cf8.slice/crio-49621b8c0786e57e4b32fb3c789958bfb5c60034fd446e18f47a593e4b02b5e3 WatchSource:0}: Error finding container 49621b8c0786e57e4b32fb3c789958bfb5c60034fd446e18f47a593e4b02b5e3: Status 404 returned error can't find the container with id 49621b8c0786e57e4b32fb3c789958bfb5c60034fd446e18f47a593e4b02b5e3
Jul 14 01:07:23.615594 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:23.615559    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7lw6w" event={"ID":"1408ccdc-23f8-4374-a582-13cc9bf26cf8","Type":"ContainerStarted","Data":"49621b8c0786e57e4b32fb3c789958bfb5c60034fd446e18f47a593e4b02b5e3"}
Jul 14 01:07:24.621029 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:24.620990    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7lw6w" event={"ID":"1408ccdc-23f8-4374-a582-13cc9bf26cf8","Type":"ContainerStarted","Data":"387ea3940ff2647326c126396814217296c5ace1ccd1d17e666e3e56f4a801b3"}
Jul 14 01:07:24.640384 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:24.640326    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7lw6w" podStartSLOduration=1.182510102 podStartE2EDuration="2.640308921s" totalImagesPullingTime="1.457798819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:07:24.638505441 +0000 UTC m=+474.202914520" watchObservedRunningTime="2026-07-14 01:07:24.640308921 +0000 UTC m=+474.204717976"
Jul 14 01:07:25.000481 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:25.000398    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:25.108985 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:25.108928    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:25.929236 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:25.929201    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:26.038914 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:26.038881    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:27.932066 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:27.932036    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:28.397642 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:28.397608    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:28.656752 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:28.656666    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8tnvj"]
Jul 14 01:07:28.658835 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:28.658817    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tnvj"
Jul 14 01:07:28.660776 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:28.660757    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8vtpm\""
Jul 14 01:07:28.660865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:28.660760    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 14 01:07:28.661312 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:28.661290    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 14 01:07:28.669627 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:28.669600    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8tnvj"]
Jul 14 01:07:28.755242 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:28.755202    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhkk4\" (UniqueName: \"kubernetes.io/projected/ffa7b2fb-d691-454d-8499-ec02afad36e9-kube-api-access-fhkk4\") pod \"cert-manager-cainjector-559cdd8c87-8tnvj\" (UID: \"ffa7b2fb-d691-454d-8499-ec02afad36e9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tnvj"
Jul 14 01:07:28.755242 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:28.755244    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ffa7b2fb-d691-454d-8499-ec02afad36e9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8tnvj\" (UID: \"ffa7b2fb-d691-454d-8499-ec02afad36e9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tnvj"
Jul 14 01:07:28.855688 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:28.855655    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhkk4\" (UniqueName: \"kubernetes.io/projected/ffa7b2fb-d691-454d-8499-ec02afad36e9-kube-api-access-fhkk4\") pod \"cert-manager-cainjector-559cdd8c87-8tnvj\" (UID: \"ffa7b2fb-d691-454d-8499-ec02afad36e9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tnvj"
Jul 14 01:07:28.855688 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:28.855693    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ffa7b2fb-d691-454d-8499-ec02afad36e9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8tnvj\" (UID: \"ffa7b2fb-d691-454d-8499-ec02afad36e9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tnvj"
Jul 14 01:07:28.863255 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:28.863224    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ffa7b2fb-d691-454d-8499-ec02afad36e9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-8tnvj\" (UID: \"ffa7b2fb-d691-454d-8499-ec02afad36e9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tnvj"
Jul 14 01:07:28.863389 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:28.863311    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhkk4\" (UniqueName: \"kubernetes.io/projected/ffa7b2fb-d691-454d-8499-ec02afad36e9-kube-api-access-fhkk4\") pod \"cert-manager-cainjector-559cdd8c87-8tnvj\" (UID: \"ffa7b2fb-d691-454d-8499-ec02afad36e9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tnvj"
Jul 14 01:07:28.973984 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:28.973889    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tnvj"
Jul 14 01:07:29.107023 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:29.106987    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-8tnvj"]
Jul 14 01:07:29.111384 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:07:29.111344    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffa7b2fb_d691_454d_8499_ec02afad36e9.slice/crio-ac795dd74d8d6c62882e92e485ff7026f2c44695c6cc75f29d9ac07dfb4991b1 WatchSource:0}: Error finding container ac795dd74d8d6c62882e92e485ff7026f2c44695c6cc75f29d9ac07dfb4991b1: Status 404 returned error can't find the container with id ac795dd74d8d6c62882e92e485ff7026f2c44695c6cc75f29d9ac07dfb4991b1
Jul 14 01:07:29.641861 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:29.641820    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tnvj" event={"ID":"ffa7b2fb-d691-454d-8499-ec02afad36e9","Type":"ContainerStarted","Data":"ac795dd74d8d6c62882e92e485ff7026f2c44695c6cc75f29d9ac07dfb4991b1"}
Jul 14 01:07:30.123625 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.123588    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"]
Jul 14 01:07:30.126249 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.126229    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"
Jul 14 01:07:30.128153 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.128132    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 14 01:07:30.128617 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.128597    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hrggj\""
Jul 14 01:07:30.128714 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.128605    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 14 01:07:30.136383 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.136361    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"]
Jul 14 01:07:30.265819 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.265781    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30a08182-8494-4fd8-b8a3-61d25726fab6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6\" (UID: \"30a08182-8494-4fd8-b8a3-61d25726fab6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"
Jul 14 01:07:30.266003 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.265835    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30a08182-8494-4fd8-b8a3-61d25726fab6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6\" (UID: \"30a08182-8494-4fd8-b8a3-61d25726fab6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"
Jul 14 01:07:30.266003 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.265881    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g27mf\" (UniqueName: \"kubernetes.io/projected/30a08182-8494-4fd8-b8a3-61d25726fab6-kube-api-access-g27mf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6\" (UID: \"30a08182-8494-4fd8-b8a3-61d25726fab6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"
Jul 14 01:07:30.367167 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.367131    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30a08182-8494-4fd8-b8a3-61d25726fab6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6\" (UID: \"30a08182-8494-4fd8-b8a3-61d25726fab6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"
Jul 14 01:07:30.367307 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.367184    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30a08182-8494-4fd8-b8a3-61d25726fab6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6\" (UID: \"30a08182-8494-4fd8-b8a3-61d25726fab6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"
Jul 14 01:07:30.367362 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.367309    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g27mf\" (UniqueName: \"kubernetes.io/projected/30a08182-8494-4fd8-b8a3-61d25726fab6-kube-api-access-g27mf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6\" (UID: \"30a08182-8494-4fd8-b8a3-61d25726fab6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"
Jul 14 01:07:30.367535 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.367518    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30a08182-8494-4fd8-b8a3-61d25726fab6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6\" (UID: \"30a08182-8494-4fd8-b8a3-61d25726fab6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"
Jul 14 01:07:30.367578 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.367547    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30a08182-8494-4fd8-b8a3-61d25726fab6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6\" (UID: \"30a08182-8494-4fd8-b8a3-61d25726fab6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"
Jul 14 01:07:30.374658 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.374599    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g27mf\" (UniqueName: \"kubernetes.io/projected/30a08182-8494-4fd8-b8a3-61d25726fab6-kube-api-access-g27mf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6\" (UID: \"30a08182-8494-4fd8-b8a3-61d25726fab6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"
Jul 14 01:07:30.436322 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.436281    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"
Jul 14 01:07:30.569041 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.569001    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"]
Jul 14 01:07:30.571662 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:07:30.571636    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a08182_8494_4fd8_b8a3_61d25726fab6.slice/crio-9abc42a4e192d128b93e555f028607b9c38029d29a07464c5edda4db35be3890 WatchSource:0}: Error finding container 9abc42a4e192d128b93e555f028607b9c38029d29a07464c5edda4db35be3890: Status 404 returned error can't find the container with id 9abc42a4e192d128b93e555f028607b9c38029d29a07464c5edda4db35be3890
Jul 14 01:07:30.648418 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.648382    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6" event={"ID":"30a08182-8494-4fd8-b8a3-61d25726fab6","Type":"ContainerStarted","Data":"720693d53faba541d1da8200a3aa0c4e431bd7200fff99bc765c61cb1151d072"}
Jul 14 01:07:30.648590 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.648445    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6" event={"ID":"30a08182-8494-4fd8-b8a3-61d25726fab6","Type":"ContainerStarted","Data":"9abc42a4e192d128b93e555f028607b9c38029d29a07464c5edda4db35be3890"}
Jul 14 01:07:30.702111 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:30.702076    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:31.654054 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:31.653938    2567 generic.go:356] "Generic (PLEG): container finished" podID="30a08182-8494-4fd8-b8a3-61d25726fab6" containerID="720693d53faba541d1da8200a3aa0c4e431bd7200fff99bc765c61cb1151d072" exitCode=0
Jul 14 01:07:31.654054 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:31.654027    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6" event={"ID":"30a08182-8494-4fd8-b8a3-61d25726fab6","Type":"ContainerDied","Data":"720693d53faba541d1da8200a3aa0c4e431bd7200fff99bc765c61cb1151d072"}
Jul 14 01:07:32.304526 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:32.304432    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:32.404311 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:32.404278    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:34.236772 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:34.236733    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jznjq"]
Jul 14 01:07:34.239236 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:34.239210    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jznjq"
Jul 14 01:07:34.241234 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:34.241212    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-sjlgf\""
Jul 14 01:07:34.248255 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:34.248225    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jznjq"]
Jul 14 01:07:34.403314 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:34.403266    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83e64c6e-7847-4527-8bcd-8daecc26f2f8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jznjq\" (UID: \"83e64c6e-7847-4527-8bcd-8daecc26f2f8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jznjq"
Jul 14 01:07:34.403522 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:34.403339    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b54r\" (UniqueName: \"kubernetes.io/projected/83e64c6e-7847-4527-8bcd-8daecc26f2f8-kube-api-access-4b54r\") pod \"cert-manager-webhook-6486ffd997-jznjq\" (UID: \"83e64c6e-7847-4527-8bcd-8daecc26f2f8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jznjq"
Jul 14 01:07:34.504293 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:34.504206    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83e64c6e-7847-4527-8bcd-8daecc26f2f8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jznjq\" (UID: \"83e64c6e-7847-4527-8bcd-8daecc26f2f8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jznjq"
Jul 14 01:07:34.504293 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:34.504274    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b54r\" (UniqueName: \"kubernetes.io/projected/83e64c6e-7847-4527-8bcd-8daecc26f2f8-kube-api-access-4b54r\") pod \"cert-manager-webhook-6486ffd997-jznjq\" (UID: \"83e64c6e-7847-4527-8bcd-8daecc26f2f8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jznjq"
Jul 14 01:07:34.512514 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:34.512484    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83e64c6e-7847-4527-8bcd-8daecc26f2f8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jznjq\" (UID: \"83e64c6e-7847-4527-8bcd-8daecc26f2f8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jznjq"
Jul 14 01:07:34.512668 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:34.512546    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b54r\" (UniqueName: \"kubernetes.io/projected/83e64c6e-7847-4527-8bcd-8daecc26f2f8-kube-api-access-4b54r\") pod \"cert-manager-webhook-6486ffd997-jznjq\" (UID: \"83e64c6e-7847-4527-8bcd-8daecc26f2f8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jznjq"
Jul 14 01:07:34.549442 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:34.549409    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jznjq"
Jul 14 01:07:35.076370 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:35.076330    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jznjq"]
Jul 14 01:07:35.089829 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:07:35.089793    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83e64c6e_7847_4527_8bcd_8daecc26f2f8.slice/crio-4780017ae40c043b2abd90537728ff2bb45475ce55149f156e97084ce4b3e568 WatchSource:0}: Error finding container 4780017ae40c043b2abd90537728ff2bb45475ce55149f156e97084ce4b3e568: Status 404 returned error can't find the container with id 4780017ae40c043b2abd90537728ff2bb45475ce55149f156e97084ce4b3e568
Jul 14 01:07:35.673795 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:35.673753    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jznjq" event={"ID":"83e64c6e-7847-4527-8bcd-8daecc26f2f8","Type":"ContainerStarted","Data":"03163441fc0731c7202ad0b79fe2aa1c211eee989f94f02b5540a044df68cbd3"}
Jul 14 01:07:35.674294 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:35.673804    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jznjq"
Jul 14 01:07:35.674294 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:35.673823    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jznjq" event={"ID":"83e64c6e-7847-4527-8bcd-8daecc26f2f8","Type":"ContainerStarted","Data":"4780017ae40c043b2abd90537728ff2bb45475ce55149f156e97084ce4b3e568"}
Jul 14 01:07:35.675540 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:35.675505    2567 generic.go:356] "Generic (PLEG): container finished" podID="30a08182-8494-4fd8-b8a3-61d25726fab6" containerID="a85e9bb7f378c389df75ed1be19c599fb4f121ba6ae01fd644a2da0c4a4bb7bd" exitCode=0
Jul 14 01:07:35.675672 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:35.675587    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6" event={"ID":"30a08182-8494-4fd8-b8a3-61d25726fab6","Type":"ContainerDied","Data":"a85e9bb7f378c389df75ed1be19c599fb4f121ba6ae01fd644a2da0c4a4bb7bd"}
Jul 14 01:07:35.677211 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:35.677163    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tnvj" event={"ID":"ffa7b2fb-d691-454d-8499-ec02afad36e9","Type":"ContainerStarted","Data":"31bb4731997a74c1995d0f54555eea17fa96245276a4bf7e1e371e0261c6a2dd"}
Jul 14 01:07:35.688803 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:35.688754    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-jznjq" podStartSLOduration=1.6887346970000001 podStartE2EDuration="1.688734697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:07:35.688318915 +0000 UTC m=+485.252727972" watchObservedRunningTime="2026-07-14 01:07:35.688734697 +0000 UTC m=+485.253143754"
Jul 14 01:07:35.703428 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:35.703379    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-8tnvj" podStartSLOduration=1.879386664 podStartE2EDuration="7.703362419s" totalImagesPullingTime="5.823975755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:07:35.702365596 +0000 UTC m=+485.266774653" watchObservedRunningTime="2026-07-14 01:07:35.703362419 +0000 UTC m=+485.267771473"
Jul 14 01:07:35.751327 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:35.751298    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:35.851573 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:35.851524    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:36.533595 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:36.533557    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:36.683795 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:36.683763    2567 generic.go:356] "Generic (PLEG): container finished" podID="30a08182-8494-4fd8-b8a3-61d25726fab6" containerID="d15fadf652ecd7afcdf5bb1c084985297c18d1ab34e24df79226166848501ff7" exitCode=0
Jul 14 01:07:36.684229 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:36.683860    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6" event={"ID":"30a08182-8494-4fd8-b8a3-61d25726fab6","Type":"ContainerDied","Data":"d15fadf652ecd7afcdf5bb1c084985297c18d1ab34e24df79226166848501ff7"}
Jul 14 01:07:37.830734 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:37.830706    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"
Jul 14 01:07:37.933830 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:37.933794    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30a08182-8494-4fd8-b8a3-61d25726fab6-util\") pod \"30a08182-8494-4fd8-b8a3-61d25726fab6\" (UID: \"30a08182-8494-4fd8-b8a3-61d25726fab6\") "
Jul 14 01:07:37.934047 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:37.933848    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g27mf\" (UniqueName: \"kubernetes.io/projected/30a08182-8494-4fd8-b8a3-61d25726fab6-kube-api-access-g27mf\") pod \"30a08182-8494-4fd8-b8a3-61d25726fab6\" (UID: \"30a08182-8494-4fd8-b8a3-61d25726fab6\") "
Jul 14 01:07:37.934047 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:37.933876    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30a08182-8494-4fd8-b8a3-61d25726fab6-bundle\") pod \"30a08182-8494-4fd8-b8a3-61d25726fab6\" (UID: \"30a08182-8494-4fd8-b8a3-61d25726fab6\") "
Jul 14 01:07:37.934282 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:37.934257    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a08182-8494-4fd8-b8a3-61d25726fab6-bundle" (OuterVolumeSpecName: "bundle") pod "30a08182-8494-4fd8-b8a3-61d25726fab6" (UID: "30a08182-8494-4fd8-b8a3-61d25726fab6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:07:37.936146 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:37.936121    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a08182-8494-4fd8-b8a3-61d25726fab6-kube-api-access-g27mf" (OuterVolumeSpecName: "kube-api-access-g27mf") pod "30a08182-8494-4fd8-b8a3-61d25726fab6" (UID: "30a08182-8494-4fd8-b8a3-61d25726fab6"). InnerVolumeSpecName "kube-api-access-g27mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:07:37.938438 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:37.938400    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a08182-8494-4fd8-b8a3-61d25726fab6-util" (OuterVolumeSpecName: "util") pod "30a08182-8494-4fd8-b8a3-61d25726fab6" (UID: "30a08182-8494-4fd8-b8a3-61d25726fab6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:07:38.035363 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:38.035326    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30a08182-8494-4fd8-b8a3-61d25726fab6-util\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:07:38.035363 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:38.035360    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g27mf\" (UniqueName: \"kubernetes.io/projected/30a08182-8494-4fd8-b8a3-61d25726fab6-kube-api-access-g27mf\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:07:38.035560 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:38.035375    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30a08182-8494-4fd8-b8a3-61d25726fab6-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:07:38.694629 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:38.694603    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6"
Jul 14 01:07:38.694629 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:38.694623    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6t7g6" event={"ID":"30a08182-8494-4fd8-b8a3-61d25726fab6","Type":"ContainerDied","Data":"9abc42a4e192d128b93e555f028607b9c38029d29a07464c5edda4db35be3890"}
Jul 14 01:07:38.694848 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:38.694665    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9abc42a4e192d128b93e555f028607b9c38029d29a07464c5edda4db35be3890"
Jul 14 01:07:41.686357 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:41.686325    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jznjq"
Jul 14 01:07:42.384409 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:42.384375    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:42.479476 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:42.479440    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:49.940294 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:49.940243    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:50.043583 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:50.043533    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:51.590750 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.590712    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"]
Jul 14 01:07:51.591151 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.591011    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a08182-8494-4fd8-b8a3-61d25726fab6" containerName="util"
Jul 14 01:07:51.591151 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.591023    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a08182-8494-4fd8-b8a3-61d25726fab6" containerName="util"
Jul 14 01:07:51.591151 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.591034    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a08182-8494-4fd8-b8a3-61d25726fab6" containerName="pull"
Jul 14 01:07:51.591151 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.591040    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a08182-8494-4fd8-b8a3-61d25726fab6" containerName="pull"
Jul 14 01:07:51.591151 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.591055    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a08182-8494-4fd8-b8a3-61d25726fab6" containerName="extract"
Jul 14 01:07:51.591151 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.591060    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a08182-8494-4fd8-b8a3-61d25726fab6" containerName="extract"
Jul 14 01:07:51.591151 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.591125    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a08182-8494-4fd8-b8a3-61d25726fab6" containerName="extract"
Jul 14 01:07:51.593727 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.593710    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"
Jul 14 01:07:51.595588 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.595564    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hrggj\""
Jul 14 01:07:51.595588 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.595575    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 14 01:07:51.596066 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.596048    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 14 01:07:51.601422 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.601402    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"]
Jul 14 01:07:51.741891 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.741851    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtpsc\" (UniqueName: \"kubernetes.io/projected/83691ed0-eb53-483f-bbbd-dc850bc23e5b-kube-api-access-jtpsc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs\" (UID: \"83691ed0-eb53-483f-bbbd-dc850bc23e5b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"
Jul 14 01:07:51.742119 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.741918    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83691ed0-eb53-483f-bbbd-dc850bc23e5b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs\" (UID: \"83691ed0-eb53-483f-bbbd-dc850bc23e5b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"
Jul 14 01:07:51.742119 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.741988    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83691ed0-eb53-483f-bbbd-dc850bc23e5b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs\" (UID: \"83691ed0-eb53-483f-bbbd-dc850bc23e5b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"
Jul 14 01:07:51.843157 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.843049    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtpsc\" (UniqueName: \"kubernetes.io/projected/83691ed0-eb53-483f-bbbd-dc850bc23e5b-kube-api-access-jtpsc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs\" (UID: \"83691ed0-eb53-483f-bbbd-dc850bc23e5b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"
Jul 14 01:07:51.843157 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.843111    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83691ed0-eb53-483f-bbbd-dc850bc23e5b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs\" (UID: \"83691ed0-eb53-483f-bbbd-dc850bc23e5b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"
Jul 14 01:07:51.843157 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.843144    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83691ed0-eb53-483f-bbbd-dc850bc23e5b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs\" (UID: \"83691ed0-eb53-483f-bbbd-dc850bc23e5b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"
Jul 14 01:07:51.843623 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.843600    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83691ed0-eb53-483f-bbbd-dc850bc23e5b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs\" (UID: \"83691ed0-eb53-483f-bbbd-dc850bc23e5b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"
Jul 14 01:07:51.843675 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.843600    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83691ed0-eb53-483f-bbbd-dc850bc23e5b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs\" (UID: \"83691ed0-eb53-483f-bbbd-dc850bc23e5b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"
Jul 14 01:07:51.850255 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.850224    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtpsc\" (UniqueName: \"kubernetes.io/projected/83691ed0-eb53-483f-bbbd-dc850bc23e5b-kube-api-access-jtpsc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs\" (UID: \"83691ed0-eb53-483f-bbbd-dc850bc23e5b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"
Jul 14 01:07:51.903910 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:51.903866    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"
Jul 14 01:07:52.036640 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:52.036605    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"]
Jul 14 01:07:52.040292 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:07:52.040253    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83691ed0_eb53_483f_bbbd_dc850bc23e5b.slice/crio-ba76f93e673bb2728e65035e198923878baffb16c89ce02e27d39276231016c1 WatchSource:0}: Error finding container ba76f93e673bb2728e65035e198923878baffb16c89ce02e27d39276231016c1: Status 404 returned error can't find the container with id ba76f93e673bb2728e65035e198923878baffb16c89ce02e27d39276231016c1
Jul 14 01:07:52.103362 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:52.103335    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:07:52.752013 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:52.751978    2567 generic.go:356] "Generic (PLEG): container finished" podID="83691ed0-eb53-483f-bbbd-dc850bc23e5b" containerID="a4cc8d25ad680cc685eae98cb797a6c80d877a785db6e895edbfcbb42afb2d8a" exitCode=0
Jul 14 01:07:52.752013 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:52.752023    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs" event={"ID":"83691ed0-eb53-483f-bbbd-dc850bc23e5b","Type":"ContainerDied","Data":"a4cc8d25ad680cc685eae98cb797a6c80d877a785db6e895edbfcbb42afb2d8a"}
Jul 14 01:07:52.752628 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:52.752066    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs" event={"ID":"83691ed0-eb53-483f-bbbd-dc850bc23e5b","Type":"ContainerStarted","Data":"ba76f93e673bb2728e65035e198923878baffb16c89ce02e27d39276231016c1"}
Jul 14 01:07:53.758292 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:53.758192    2567 generic.go:356] "Generic (PLEG): container finished" podID="83691ed0-eb53-483f-bbbd-dc850bc23e5b" containerID="205c212cf2100a722226f04db9d0c355750ddb71ffcd8365a5b58c4dbbe60d11" exitCode=0
Jul 14 01:07:53.758292 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:53.758254    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs" event={"ID":"83691ed0-eb53-483f-bbbd-dc850bc23e5b","Type":"ContainerDied","Data":"205c212cf2100a722226f04db9d0c355750ddb71ffcd8365a5b58c4dbbe60d11"}
Jul 14 01:07:54.764691 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:54.764651    2567 generic.go:356] "Generic (PLEG): container finished" podID="83691ed0-eb53-483f-bbbd-dc850bc23e5b" containerID="4e00ecc3dbfc61854345d298948e51592872ef5f8608086970249c0733e3368c" exitCode=0
Jul 14 01:07:54.765118 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:54.764727    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs" event={"ID":"83691ed0-eb53-483f-bbbd-dc850bc23e5b","Type":"ContainerDied","Data":"4e00ecc3dbfc61854345d298948e51592872ef5f8608086970249c0733e3368c"}
Jul 14 01:07:55.899006 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:55.898977    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"
Jul 14 01:07:56.075514 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:56.075426    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83691ed0-eb53-483f-bbbd-dc850bc23e5b-bundle\") pod \"83691ed0-eb53-483f-bbbd-dc850bc23e5b\" (UID: \"83691ed0-eb53-483f-bbbd-dc850bc23e5b\") "
Jul 14 01:07:56.075514 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:56.075497    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtpsc\" (UniqueName: \"kubernetes.io/projected/83691ed0-eb53-483f-bbbd-dc850bc23e5b-kube-api-access-jtpsc\") pod \"83691ed0-eb53-483f-bbbd-dc850bc23e5b\" (UID: \"83691ed0-eb53-483f-bbbd-dc850bc23e5b\") "
Jul 14 01:07:56.075514 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:56.075518    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83691ed0-eb53-483f-bbbd-dc850bc23e5b-util\") pod \"83691ed0-eb53-483f-bbbd-dc850bc23e5b\" (UID: \"83691ed0-eb53-483f-bbbd-dc850bc23e5b\") "
Jul 14 01:07:56.076395 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:56.076365    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83691ed0-eb53-483f-bbbd-dc850bc23e5b-bundle" (OuterVolumeSpecName: "bundle") pod "83691ed0-eb53-483f-bbbd-dc850bc23e5b" (UID: "83691ed0-eb53-483f-bbbd-dc850bc23e5b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:07:56.077797 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:56.077766    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83691ed0-eb53-483f-bbbd-dc850bc23e5b-kube-api-access-jtpsc" (OuterVolumeSpecName: "kube-api-access-jtpsc") pod "83691ed0-eb53-483f-bbbd-dc850bc23e5b" (UID: "83691ed0-eb53-483f-bbbd-dc850bc23e5b"). InnerVolumeSpecName "kube-api-access-jtpsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:07:56.082111 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:56.082085    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83691ed0-eb53-483f-bbbd-dc850bc23e5b-util" (OuterVolumeSpecName: "util") pod "83691ed0-eb53-483f-bbbd-dc850bc23e5b" (UID: "83691ed0-eb53-483f-bbbd-dc850bc23e5b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:07:56.176905 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:56.176869    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83691ed0-eb53-483f-bbbd-dc850bc23e5b-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:07:56.176905 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:56.176902    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtpsc\" (UniqueName: \"kubernetes.io/projected/83691ed0-eb53-483f-bbbd-dc850bc23e5b-kube-api-access-jtpsc\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:07:56.176905 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:56.176912    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83691ed0-eb53-483f-bbbd-dc850bc23e5b-util\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:07:56.775702 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:56.775669    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs"
Jul 14 01:07:56.775702 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:56.775684    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft85xs" event={"ID":"83691ed0-eb53-483f-bbbd-dc850bc23e5b","Type":"ContainerDied","Data":"ba76f93e673bb2728e65035e198923878baffb16c89ce02e27d39276231016c1"}
Jul 14 01:07:56.775991 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:07:56.775726    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba76f93e673bb2728e65035e198923878baffb16c89ce02e27d39276231016c1"
Jul 14 01:08:02.333531 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.333493    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:02.381817 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.381786    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"]
Jul 14 01:08:02.382091 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.382078    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83691ed0-eb53-483f-bbbd-dc850bc23e5b" containerName="pull"
Jul 14 01:08:02.382145 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.382094    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="83691ed0-eb53-483f-bbbd-dc850bc23e5b" containerName="pull"
Jul 14 01:08:02.382145 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.382108    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83691ed0-eb53-483f-bbbd-dc850bc23e5b" containerName="extract"
Jul 14 01:08:02.382145 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.382114    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="83691ed0-eb53-483f-bbbd-dc850bc23e5b" containerName="extract"
Jul 14 01:08:02.382145 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.382129    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83691ed0-eb53-483f-bbbd-dc850bc23e5b" containerName="util"
Jul 14 01:08:02.382145 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.382134    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="83691ed0-eb53-483f-bbbd-dc850bc23e5b" containerName="util"
Jul 14 01:08:02.382284 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.382177    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="83691ed0-eb53-483f-bbbd-dc850bc23e5b" containerName="extract"
Jul 14 01:08:02.386086 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.386068    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:02.388259 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.388233    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 14 01:08:02.388884 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.388870    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 14 01:08:02.389047 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.389019    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 14 01:08:02.389131 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.389019    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-92tqw\""
Jul 14 01:08:02.389131 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.389106    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 14 01:08:02.389131 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.389123    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 14 01:08:02.395283 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.395251    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"]
Jul 14 01:08:02.424564 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.424530    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:02.527600 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.527558    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3669ca4-4d0f-4bbf-b811-fd26049c94a5-cert\") pod \"lws-controller-manager-78f77bcdb4-9l448\" (UID: \"b3669ca4-4d0f-4bbf-b811-fd26049c94a5\") " pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:02.527774 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.527607    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b3669ca4-4d0f-4bbf-b811-fd26049c94a5-manager-config\") pod \"lws-controller-manager-78f77bcdb4-9l448\" (UID: \"b3669ca4-4d0f-4bbf-b811-fd26049c94a5\") " pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:02.527774 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.527661    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3669ca4-4d0f-4bbf-b811-fd26049c94a5-metrics-cert\") pod \"lws-controller-manager-78f77bcdb4-9l448\" (UID: \"b3669ca4-4d0f-4bbf-b811-fd26049c94a5\") " pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:02.527774 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.527734    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxhtq\" (UniqueName: \"kubernetes.io/projected/b3669ca4-4d0f-4bbf-b811-fd26049c94a5-kube-api-access-lxhtq\") pod \"lws-controller-manager-78f77bcdb4-9l448\" (UID: \"b3669ca4-4d0f-4bbf-b811-fd26049c94a5\") " pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:02.628970 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.628915    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b3669ca4-4d0f-4bbf-b811-fd26049c94a5-manager-config\") pod \"lws-controller-manager-78f77bcdb4-9l448\" (UID: \"b3669ca4-4d0f-4bbf-b811-fd26049c94a5\") " pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:02.629101 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.629003    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3669ca4-4d0f-4bbf-b811-fd26049c94a5-metrics-cert\") pod \"lws-controller-manager-78f77bcdb4-9l448\" (UID: \"b3669ca4-4d0f-4bbf-b811-fd26049c94a5\") " pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:02.629101 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.629041    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhtq\" (UniqueName: \"kubernetes.io/projected/b3669ca4-4d0f-4bbf-b811-fd26049c94a5-kube-api-access-lxhtq\") pod \"lws-controller-manager-78f77bcdb4-9l448\" (UID: \"b3669ca4-4d0f-4bbf-b811-fd26049c94a5\") " pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:02.629101 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.629074    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3669ca4-4d0f-4bbf-b811-fd26049c94a5-cert\") pod \"lws-controller-manager-78f77bcdb4-9l448\" (UID: \"b3669ca4-4d0f-4bbf-b811-fd26049c94a5\") " pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:02.629639 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.629614    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b3669ca4-4d0f-4bbf-b811-fd26049c94a5-manager-config\") pod \"lws-controller-manager-78f77bcdb4-9l448\" (UID: \"b3669ca4-4d0f-4bbf-b811-fd26049c94a5\") " pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:02.631687 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.631663    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3669ca4-4d0f-4bbf-b811-fd26049c94a5-metrics-cert\") pod \"lws-controller-manager-78f77bcdb4-9l448\" (UID: \"b3669ca4-4d0f-4bbf-b811-fd26049c94a5\") " pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:02.631783 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.631741    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3669ca4-4d0f-4bbf-b811-fd26049c94a5-cert\") pod \"lws-controller-manager-78f77bcdb4-9l448\" (UID: \"b3669ca4-4d0f-4bbf-b811-fd26049c94a5\") " pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:02.640815 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.640784    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxhtq\" (UniqueName: \"kubernetes.io/projected/b3669ca4-4d0f-4bbf-b811-fd26049c94a5-kube-api-access-lxhtq\") pod \"lws-controller-manager-78f77bcdb4-9l448\" (UID: \"b3669ca4-4d0f-4bbf-b811-fd26049c94a5\") " pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:02.697174 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.697128    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:02.835305 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:02.835274    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"]
Jul 14 01:08:02.838172 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:08:02.838139    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3669ca4_4d0f_4bbf_b811_fd26049c94a5.slice/crio-d83358cdb0f295248e7a20b829702df23ee85a4b6a1f5efe8fcbf318a7f5ea2b WatchSource:0}: Error finding container d83358cdb0f295248e7a20b829702df23ee85a4b6a1f5efe8fcbf318a7f5ea2b: Status 404 returned error can't find the container with id d83358cdb0f295248e7a20b829702df23ee85a4b6a1f5efe8fcbf318a7f5ea2b
Jul 14 01:08:03.807037 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:03.806994    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448" event={"ID":"b3669ca4-4d0f-4bbf-b811-fd26049c94a5","Type":"ContainerStarted","Data":"d83358cdb0f295248e7a20b829702df23ee85a4b6a1f5efe8fcbf318a7f5ea2b"}
Jul 14 01:08:07.751777 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:07.751742    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:07.843526 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:07.843495    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:08.681062 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.681027    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"]
Jul 14 01:08:08.685776 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.685748    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.688400 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.688371    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 14 01:08:08.688400 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.688395    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 14 01:08:08.688612 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.688378    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 14 01:08:08.688877 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.688859    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fbcld\""
Jul 14 01:08:08.688932 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.688882    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 14 01:08:08.700596 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.700561    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"]
Jul 14 01:08:08.769512 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.769477    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/305439f7-e716-4f07-b05f-de32c0814257-manifests\") pod \"opendatahub-operator-controller-manager-bc876bb4c-cpstz\" (UID: \"305439f7-e716-4f07-b05f-de32c0814257\") " pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.769897 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.769518    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdbw8\" (UniqueName: \"kubernetes.io/projected/305439f7-e716-4f07-b05f-de32c0814257-kube-api-access-tdbw8\") pod \"opendatahub-operator-controller-manager-bc876bb4c-cpstz\" (UID: \"305439f7-e716-4f07-b05f-de32c0814257\") " pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.769897 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.769544    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/305439f7-e716-4f07-b05f-de32c0814257-tmp\") pod \"opendatahub-operator-controller-manager-bc876bb4c-cpstz\" (UID: \"305439f7-e716-4f07-b05f-de32c0814257\") " pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.769897 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.769613    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/305439f7-e716-4f07-b05f-de32c0814257-webhook-cert\") pod \"opendatahub-operator-controller-manager-bc876bb4c-cpstz\" (UID: \"305439f7-e716-4f07-b05f-de32c0814257\") " pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.769897 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.769641    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/305439f7-e716-4f07-b05f-de32c0814257-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bc876bb4c-cpstz\" (UID: \"305439f7-e716-4f07-b05f-de32c0814257\") " pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.870234 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.870194    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/305439f7-e716-4f07-b05f-de32c0814257-manifests\") pod \"opendatahub-operator-controller-manager-bc876bb4c-cpstz\" (UID: \"305439f7-e716-4f07-b05f-de32c0814257\") " pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.870234 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.870240    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdbw8\" (UniqueName: \"kubernetes.io/projected/305439f7-e716-4f07-b05f-de32c0814257-kube-api-access-tdbw8\") pod \"opendatahub-operator-controller-manager-bc876bb4c-cpstz\" (UID: \"305439f7-e716-4f07-b05f-de32c0814257\") " pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.870452 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.870368    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/305439f7-e716-4f07-b05f-de32c0814257-tmp\") pod \"opendatahub-operator-controller-manager-bc876bb4c-cpstz\" (UID: \"305439f7-e716-4f07-b05f-de32c0814257\") " pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.870452 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.870438    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/305439f7-e716-4f07-b05f-de32c0814257-webhook-cert\") pod \"opendatahub-operator-controller-manager-bc876bb4c-cpstz\" (UID: \"305439f7-e716-4f07-b05f-de32c0814257\") " pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.870559 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.870468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/305439f7-e716-4f07-b05f-de32c0814257-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bc876bb4c-cpstz\" (UID: \"305439f7-e716-4f07-b05f-de32c0814257\") " pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.870652 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.870630    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/305439f7-e716-4f07-b05f-de32c0814257-manifests\") pod \"opendatahub-operator-controller-manager-bc876bb4c-cpstz\" (UID: \"305439f7-e716-4f07-b05f-de32c0814257\") " pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.872792 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.872764    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/305439f7-e716-4f07-b05f-de32c0814257-tmp\") pod \"opendatahub-operator-controller-manager-bc876bb4c-cpstz\" (UID: \"305439f7-e716-4f07-b05f-de32c0814257\") " pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.872927 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.872911    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/305439f7-e716-4f07-b05f-de32c0814257-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bc876bb4c-cpstz\" (UID: \"305439f7-e716-4f07-b05f-de32c0814257\") " pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.873085 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.873066    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/305439f7-e716-4f07-b05f-de32c0814257-webhook-cert\") pod \"opendatahub-operator-controller-manager-bc876bb4c-cpstz\" (UID: \"305439f7-e716-4f07-b05f-de32c0814257\") " pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.896165 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.896136    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdbw8\" (UniqueName: \"kubernetes.io/projected/305439f7-e716-4f07-b05f-de32c0814257-kube-api-access-tdbw8\") pod \"opendatahub-operator-controller-manager-bc876bb4c-cpstz\" (UID: \"305439f7-e716-4f07-b05f-de32c0814257\") " pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:08.905225 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.905197    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:08.996330 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:08.996234    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:09.004207 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:09.004174    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:09.107582 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:09.107545    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:09.142730 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:09.142703    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"]
Jul 14 01:08:09.145790 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:08:09.145761    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305439f7_e716_4f07_b05f_de32c0814257.slice/crio-996114391d9430da21400705adbed0ec4d8283a4bddfff302fb0f741e18e894a WatchSource:0}: Error finding container 996114391d9430da21400705adbed0ec4d8283a4bddfff302fb0f741e18e894a: Status 404 returned error can't find the container with id 996114391d9430da21400705adbed0ec4d8283a4bddfff302fb0f741e18e894a
Jul 14 01:08:09.211794 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:09.211756    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:09.306256 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:09.306156    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:09.406460 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:09.406427    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:09.834140 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:09.834098    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz" event={"ID":"305439f7-e716-4f07-b05f-de32c0814257","Type":"ContainerStarted","Data":"996114391d9430da21400705adbed0ec4d8283a4bddfff302fb0f741e18e894a"}
Jul 14 01:08:12.851119 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:12.851081    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz" event={"ID":"305439f7-e716-4f07-b05f-de32c0814257","Type":"ContainerStarted","Data":"4b54be756b4db496c272f9e2fa1eaa716de51e9f0fe8d9dcfec527cb88f1299d"}
Jul 14 01:08:14.863438 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:14.863396    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448" event={"ID":"b3669ca4-4d0f-4bbf-b811-fd26049c94a5","Type":"ContainerStarted","Data":"4aaebee25fa1fd2814162e722657ac75e62a67d2631608fefc74a3e651cf91f3"}
Jul 14 01:08:14.863899 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:14.863471    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:14.865030 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:14.865006    2567 generic.go:356] "Generic (PLEG): container finished" podID="305439f7-e716-4f07-b05f-de32c0814257" containerID="4b54be756b4db496c272f9e2fa1eaa716de51e9f0fe8d9dcfec527cb88f1299d" exitCode=0
Jul 14 01:08:14.865117 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:14.865048    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz" event={"ID":"305439f7-e716-4f07-b05f-de32c0814257","Type":"ContainerDied","Data":"4b54be756b4db496c272f9e2fa1eaa716de51e9f0fe8d9dcfec527cb88f1299d"}
Jul 14 01:08:14.879332 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:14.879269    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448" podStartSLOduration=1.356323273 podStartE2EDuration="12.879251318s" totalImagesPullingTime="11.522928045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:08:14.878006653 +0000 UTC m=+524.442415711" watchObservedRunningTime="2026-07-14 01:08:14.879251318 +0000 UTC m=+524.443660372"
Jul 14 01:08:15.871265 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:15.871227    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz" event={"ID":"305439f7-e716-4f07-b05f-de32c0814257","Type":"ContainerStarted","Data":"56af26ee34c169d6e6249b43ec1f36132ad7a4ce3de78e032f8dedea87020d43"}
Jul 14 01:08:15.871697 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:15.871542    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:15.892705 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:15.892642    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz" podStartSLOduration=2.742978188 podStartE2EDuration="7.892619661s" totalImagesPullingTime="4.024595632s" totalInitContainerRuntime="1.125045841s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:08:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:08:15.891282571 +0000 UTC m=+525.455691625" watchObservedRunningTime="2026-07-14 01:08:15.892619661 +0000 UTC m=+525.457028718"
Jul 14 01:08:25.873688 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:25.873604    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-78f77bcdb4-9l448"
Jul 14 01:08:26.877976 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:26.877935    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-bc876bb4c-cpstz"
Jul 14 01:08:28.385065 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:28.385020    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:28.549170 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:28.549116    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:28.851602 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:28.851493    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:28.939131 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:28.939098    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:29.370133 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:29.370066    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:29.470130 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:29.470094    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:29.585827 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:29.585774    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:29.700251 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:29.700144    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:30.105714 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.105622    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:30.188747 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.188716    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tlqk9"]
Jul 14 01:08:30.192135 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.192115    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tlqk9"
Jul 14 01:08:30.194201 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.194171    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 14 01:08:30.194613 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.194568    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:30.205175 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.205150    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tlqk9"]
Jul 14 01:08:30.248090 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.248043    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92831026-a30e-45e9-8d0d-ec83e8d50819-tls-certs\") pod \"model-serving-api-54d89bd5fc-tlqk9\" (UID: \"92831026-a30e-45e9-8d0d-ec83e8d50819\") " pod="opendatahub/model-serving-api-54d89bd5fc-tlqk9"
Jul 14 01:08:30.248284 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.248142    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgfhz\" (UniqueName: \"kubernetes.io/projected/92831026-a30e-45e9-8d0d-ec83e8d50819-kube-api-access-tgfhz\") pod \"model-serving-api-54d89bd5fc-tlqk9\" (UID: \"92831026-a30e-45e9-8d0d-ec83e8d50819\") " pod="opendatahub/model-serving-api-54d89bd5fc-tlqk9"
Jul 14 01:08:30.270223 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.270190    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:30.337134 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.337101    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:30.349277 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.349243    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92831026-a30e-45e9-8d0d-ec83e8d50819-tls-certs\") pod \"model-serving-api-54d89bd5fc-tlqk9\" (UID: \"92831026-a30e-45e9-8d0d-ec83e8d50819\") " pod="opendatahub/model-serving-api-54d89bd5fc-tlqk9"
Jul 14 01:08:30.349424 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.349294    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfhz\" (UniqueName: \"kubernetes.io/projected/92831026-a30e-45e9-8d0d-ec83e8d50819-kube-api-access-tgfhz\") pod \"model-serving-api-54d89bd5fc-tlqk9\" (UID: \"92831026-a30e-45e9-8d0d-ec83e8d50819\") " pod="opendatahub/model-serving-api-54d89bd5fc-tlqk9"
Jul 14 01:08:30.351854 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.351826    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92831026-a30e-45e9-8d0d-ec83e8d50819-tls-certs\") pod \"model-serving-api-54d89bd5fc-tlqk9\" (UID: \"92831026-a30e-45e9-8d0d-ec83e8d50819\") " pod="opendatahub/model-serving-api-54d89bd5fc-tlqk9"
Jul 14 01:08:30.356746 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.356688    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgfhz\" (UniqueName: \"kubernetes.io/projected/92831026-a30e-45e9-8d0d-ec83e8d50819-kube-api-access-tgfhz\") pod \"model-serving-api-54d89bd5fc-tlqk9\" (UID: \"92831026-a30e-45e9-8d0d-ec83e8d50819\") " pod="opendatahub/model-serving-api-54d89bd5fc-tlqk9"
Jul 14 01:08:30.501911 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.501874    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tlqk9"
Jul 14 01:08:30.658148 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:08:30.658114    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92831026_a30e_45e9_8d0d_ec83e8d50819.slice/crio-e82a2c50088af9085f55941fd5ef0a053a8447fd1bda715b750c53e7c7f3da28 WatchSource:0}: Error finding container e82a2c50088af9085f55941fd5ef0a053a8447fd1bda715b750c53e7c7f3da28: Status 404 returned error can't find the container with id e82a2c50088af9085f55941fd5ef0a053a8447fd1bda715b750c53e7c7f3da28
Jul 14 01:08:30.659779 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.659652    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tlqk9"]
Jul 14 01:08:30.939974 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.939851    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tlqk9" event={"ID":"92831026-a30e-45e9-8d0d-ec83e8d50819","Type":"ContainerStarted","Data":"e82a2c50088af9085f55941fd5ef0a053a8447fd1bda715b750c53e7c7f3da28"}
Jul 14 01:08:30.943005 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:30.942948    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:31.153365 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:31.153334    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:32.950666 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:32.950577    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tlqk9" event={"ID":"92831026-a30e-45e9-8d0d-ec83e8d50819","Type":"ContainerStarted","Data":"c7052f592df79fb75ed771a82d740bae57d201eafdb41d8ff12dff1f3858583f"}
Jul 14 01:08:32.951063 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:32.950850    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-tlqk9"
Jul 14 01:08:32.966227 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:32.966170    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-tlqk9" podStartSLOduration=0.999674362 podStartE2EDuration="2.96615496s" totalImagesPullingTime="1.966480598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:08:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:08:32.964175316 +0000 UTC m=+542.528584374" watchObservedRunningTime="2026-07-14 01:08:32.96615496 +0000 UTC m=+542.530564014"
Jul 14 01:08:34.233390 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:34.233353    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:35.960235 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:35.960199    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:36.029458 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.029421    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:36.592949 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.592912    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"]
Jul 14 01:08:36.596583 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.596558    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.598657 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.598627    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 01:08:36.598820 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.598711    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hphgg\""
Jul 14 01:08:36.607492 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.607462    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"]
Jul 14 01:08:36.626106 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.626067    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:36.675822 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.675784    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"]
Jul 14 01:08:36.678884 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.678863    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.681184 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.681152    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 14 01:08:36.681184 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.681161    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 14 01:08:36.681399 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.681213    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vv9bj\""
Jul 14 01:08:36.681635 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.681614    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 14 01:08:36.691088 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.691057    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"]
Jul 14 01:08:36.703143 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.703101    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce796ea-a468-4538-9196-5fd3d87e0f15-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.703143 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.703145    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce796ea-a468-4538-9196-5fd3d87e0f15-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.703355 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.703227    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce796ea-a468-4538-9196-5fd3d87e0f15-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.703355 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.703307    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce796ea-a468-4538-9196-5fd3d87e0f15-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.703355 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.703332    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce796ea-a468-4538-9196-5fd3d87e0f15-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.703355 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.703354    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce796ea-a468-4538-9196-5fd3d87e0f15-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.703515 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.703379    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce796ea-a468-4538-9196-5fd3d87e0f15-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.703515 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.703400    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce796ea-a468-4538-9196-5fd3d87e0f15-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.703515 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.703422    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kgxt\" (UniqueName: \"kubernetes.io/projected/1ce796ea-a468-4538-9196-5fd3d87e0f15-kube-api-access-7kgxt\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.804144 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804103    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.804144 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804160    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tlbs\" (UniqueName: \"kubernetes.io/projected/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-kube-api-access-9tlbs\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.804387 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804188    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.804387 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804211    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.804387 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804259    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.804387 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804311    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce796ea-a468-4538-9196-5fd3d87e0f15-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.804387 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804335    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce796ea-a468-4538-9196-5fd3d87e0f15-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.804387 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804357    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce796ea-a468-4538-9196-5fd3d87e0f15-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.804387 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804376    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce796ea-a468-4538-9196-5fd3d87e0f15-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.804704 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804393    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.804704 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804413    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce796ea-a468-4538-9196-5fd3d87e0f15-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.804704 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804445    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kgxt\" (UniqueName: \"kubernetes.io/projected/1ce796ea-a468-4538-9196-5fd3d87e0f15-kube-api-access-7kgxt\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.804704 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804498    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce796ea-a468-4538-9196-5fd3d87e0f15-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.804704 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804537    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce796ea-a468-4538-9196-5fd3d87e0f15-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.804704 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804573    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.804704 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804617    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce796ea-a468-4538-9196-5fd3d87e0f15-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.805082 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804745    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce796ea-a468-4538-9196-5fd3d87e0f15-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.805082 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804827    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce796ea-a468-4538-9196-5fd3d87e0f15-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.805082 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.804942    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce796ea-a468-4538-9196-5fd3d87e0f15-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.805082 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.805038    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce796ea-a468-4538-9196-5fd3d87e0f15-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.805444 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.805209    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce796ea-a468-4538-9196-5fd3d87e0f15-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.806996 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.806945    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce796ea-a468-4538-9196-5fd3d87e0f15-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.807194 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.807179    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce796ea-a468-4538-9196-5fd3d87e0f15-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.816820 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.816784    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kgxt\" (UniqueName: \"kubernetes.io/projected/1ce796ea-a468-4538-9196-5fd3d87e0f15-kube-api-access-7kgxt\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.816998 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.816909    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:36.819175 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.819155    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce796ea-a468-4538-9196-5fd3d87e0f15-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vgbd2\" (UID: \"1ce796ea-a468-4538-9196-5fd3d87e0f15\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.905585 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.905539    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.905585 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.905595    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.905845 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.905616    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tlbs\" (UniqueName: \"kubernetes.io/projected/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-kube-api-access-9tlbs\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.905845 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.905748    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.905845 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.905782    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.905845 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.905805    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.906090 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.905873    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.906611 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.906583    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.907429 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.907401    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:36.908182 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.908153    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.908684 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.908661    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.908779 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.908755    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.908779 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.908763    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.918201 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.918170    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tlbs\" (UniqueName: \"kubernetes.io/projected/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-kube-api-access-9tlbs\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.920218 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.920154    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34303f0d-42bf-4c06-ab86-2b87ba91a3f6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-tdz6p\" (UID: \"34303f0d-42bf-4c06-ab86-2b87ba91a3f6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:36.948579 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.948545    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:36.988811 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:36.988780    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:37.069605 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:37.069581    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"]
Jul 14 01:08:37.071795 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:08:37.071761    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce796ea_a468_4538_9196_5fd3d87e0f15.slice/crio-7135f4b620df22d0dc90ceff4b2b66731e6e5077482ffdac523d49316efbf751 WatchSource:0}: Error finding container 7135f4b620df22d0dc90ceff4b2b66731e6e5077482ffdac523d49316efbf751: Status 404 returned error can't find the container with id 7135f4b620df22d0dc90ceff4b2b66731e6e5077482ffdac523d49316efbf751
Jul 14 01:08:37.137520 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:37.137430    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"]
Jul 14 01:08:37.140183 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:08:37.140147    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34303f0d_42bf_4c06_ab86_2b87ba91a3f6.slice/crio-30940ccb824821ac22719964c4e36e0ff7f7d950885aeba4b10f24fb80178b36 WatchSource:0}: Error finding container 30940ccb824821ac22719964c4e36e0ff7f7d950885aeba4b10f24fb80178b36: Status 404 returned error can't find the container with id 30940ccb824821ac22719964c4e36e0ff7f7d950885aeba4b10f24fb80178b36
Jul 14 01:08:37.979941 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:37.979844    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p" event={"ID":"34303f0d-42bf-4c06-ab86-2b87ba91a3f6","Type":"ContainerStarted","Data":"30940ccb824821ac22719964c4e36e0ff7f7d950885aeba4b10f24fb80178b36"}
Jul 14 01:08:37.982900 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:37.982824    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2" event={"ID":"1ce796ea-a468-4538-9196-5fd3d87e0f15","Type":"ContainerStarted","Data":"7135f4b620df22d0dc90ceff4b2b66731e6e5077482ffdac523d49316efbf751"}
Jul 14 01:08:39.380406 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:39.380368    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:39.502349 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:39.502298    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:39.589928 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:39.589855    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:39.812665 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:39.812584    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:40.685427 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:40.685387    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:40.786758 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:40.786716    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:40.847539 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:40.847498    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:08:40.847674 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:40.847574    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:08:40.851697 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:40.851669    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:08:40.851786 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:40.851732    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:08:40.851786 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:40.851761    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:08:40.930612 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:40.930569    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:40.998787 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:40.998681    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2" event={"ID":"1ce796ea-a468-4538-9196-5fd3d87e0f15","Type":"ContainerStarted","Data":"113c6582e0e39b63eeb291ea3563c8d4ccd3cc718add850bc2e83ea8a64b14bf"}
Jul 14 01:08:41.000307 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:41.000279    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p" event={"ID":"34303f0d-42bf-4c06-ab86-2b87ba91a3f6","Type":"ContainerStarted","Data":"c5f123f829cccc40e71cba70e8210803fe5d429e6b9cbbf5a98e72b99e42716a"}
Jul 14 01:08:41.000491 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:41.000470    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:41.001997 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:41.001976    2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-tdz6p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 01:08:41.002105 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:41.002023    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p" podUID="34303f0d-42bf-4c06-ab86-2b87ba91a3f6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 01:08:41.022184 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:41.022137    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2" podStartSLOduration=1.244703573 podStartE2EDuration="5.02211983s" totalImagesPullingTime="3.777416257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:08:41.020125271 +0000 UTC m=+550.584534327" watchObservedRunningTime="2026-07-14 01:08:41.02211983 +0000 UTC m=+550.586528885"
Jul 14 01:08:41.043122 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:41.043066    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p" podStartSLOduration=1.338021331 podStartE2EDuration="5.043047682s" totalImagesPullingTime="3.705026351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:08:41.040684114 +0000 UTC m=+550.605093169" watchObservedRunningTime="2026-07-14 01:08:41.043047682 +0000 UTC m=+550.607456737"
Jul 14 01:08:41.908147 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:41.908097    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:41.912885 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:41.912855    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:42.004857 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:42.004810    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:42.005452 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:42.005434    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-tdz6p"
Jul 14 01:08:42.005795 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:42.005779    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vgbd2"
Jul 14 01:08:43.702809 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:43.702767    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:08:43.959262 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:08:43.959175    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-tlqk9"
Jul 14 01:09:18.677805 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:18.677763    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:09:30.245621 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:30.245586    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:09:30.303495 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:30.303458    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:09:30.927631 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:30.927589    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:09:31.118567 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:31.118528    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:09:45.911789 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:45.911744    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"]
Jul 14 01:09:45.914557 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:45.914532    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"
Jul 14 01:09:45.916618 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:45.916590    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 14 01:09:45.916720 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:45.916619    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 14 01:09:45.916720 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:45.916707    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9l8cc\""
Jul 14 01:09:45.923223 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:45.923195    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"]
Jul 14 01:09:46.008135 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.008091    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wm5p\" (UniqueName: \"kubernetes.io/projected/f519d49d-6e68-448b-b529-981875cdc8cc-kube-api-access-6wm5p\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4\" (UID: \"f519d49d-6e68-448b-b529-981875cdc8cc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"
Jul 14 01:09:46.008311 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.008188    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f519d49d-6e68-448b-b529-981875cdc8cc-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4\" (UID: \"f519d49d-6e68-448b-b529-981875cdc8cc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"
Jul 14 01:09:46.008311 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.008218    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f519d49d-6e68-448b-b529-981875cdc8cc-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4\" (UID: \"f519d49d-6e68-448b-b529-981875cdc8cc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"
Jul 14 01:09:46.108936 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.108899    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f519d49d-6e68-448b-b529-981875cdc8cc-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4\" (UID: \"f519d49d-6e68-448b-b529-981875cdc8cc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"
Jul 14 01:09:46.108936 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.108936    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f519d49d-6e68-448b-b529-981875cdc8cc-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4\" (UID: \"f519d49d-6e68-448b-b529-981875cdc8cc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"
Jul 14 01:09:46.109205 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.109007    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wm5p\" (UniqueName: \"kubernetes.io/projected/f519d49d-6e68-448b-b529-981875cdc8cc-kube-api-access-6wm5p\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4\" (UID: \"f519d49d-6e68-448b-b529-981875cdc8cc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"
Jul 14 01:09:46.109381 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.109355    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f519d49d-6e68-448b-b529-981875cdc8cc-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4\" (UID: \"f519d49d-6e68-448b-b529-981875cdc8cc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"
Jul 14 01:09:46.109595 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.109575    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f519d49d-6e68-448b-b529-981875cdc8cc-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4\" (UID: \"f519d49d-6e68-448b-b529-981875cdc8cc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"
Jul 14 01:09:46.116801 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.116767    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wm5p\" (UniqueName: \"kubernetes.io/projected/f519d49d-6e68-448b-b529-981875cdc8cc-kube-api-access-6wm5p\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4\" (UID: \"f519d49d-6e68-448b-b529-981875cdc8cc\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"
Jul 14 01:09:46.223788 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.223696    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"
Jul 14 01:09:46.363111 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.363083    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"]
Jul 14 01:09:46.365694 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:09:46.365667    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf519d49d_6e68_448b_b529_981875cdc8cc.slice/crio-79f72310d267737c304c3ec04cb8003131a89b61b56c301f05ddbfd70d155987 WatchSource:0}: Error finding container 79f72310d267737c304c3ec04cb8003131a89b61b56c301f05ddbfd70d155987: Status 404 returned error can't find the container with id 79f72310d267737c304c3ec04cb8003131a89b61b56c301f05ddbfd70d155987
Jul 14 01:09:46.709101 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.709068    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"]
Jul 14 01:09:46.711358 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.711339    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"
Jul 14 01:09:46.719527 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.719498    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"]
Jul 14 01:09:46.814782 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.814741    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt\" (UID: \"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"
Jul 14 01:09:46.815003 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.814795    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b94t\" (UniqueName: \"kubernetes.io/projected/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-kube-api-access-8b94t\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt\" (UID: \"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"
Jul 14 01:09:46.815003 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.814881    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt\" (UID: \"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"
Jul 14 01:09:46.915713 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.915675    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt\" (UID: \"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"
Jul 14 01:09:46.916249 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.915727    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b94t\" (UniqueName: \"kubernetes.io/projected/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-kube-api-access-8b94t\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt\" (UID: \"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"
Jul 14 01:09:46.916249 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.915769    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt\" (UID: \"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"
Jul 14 01:09:46.916249 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.916218    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt\" (UID: \"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"
Jul 14 01:09:46.916249 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.916212    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt\" (UID: \"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"
Jul 14 01:09:46.923267 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:46.923240    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b94t\" (UniqueName: \"kubernetes.io/projected/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-kube-api-access-8b94t\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt\" (UID: \"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"
Jul 14 01:09:47.021816 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.021720    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"
Jul 14 01:09:47.113558 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.113521    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"]
Jul 14 01:09:47.116347 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.116319    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"
Jul 14 01:09:47.124792 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.124757    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"]
Jul 14 01:09:47.161177 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.161151    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"]
Jul 14 01:09:47.163475 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:09:47.163451    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe84fb62_ae2f_4a3e_aba4_183d1d0f4a86.slice/crio-c84c28069ee0f1785a6a86ba8042ab2df6f8227b278d87c980b9dd88a0507429 WatchSource:0}: Error finding container c84c28069ee0f1785a6a86ba8042ab2df6f8227b278d87c980b9dd88a0507429: Status 404 returned error can't find the container with id c84c28069ee0f1785a6a86ba8042ab2df6f8227b278d87c980b9dd88a0507429
Jul 14 01:09:47.219002 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.218974    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eaa6228d-8835-49c5-a0ed-77b810d4890f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s\" (UID: \"eaa6228d-8835-49c5-a0ed-77b810d4890f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"
Jul 14 01:09:47.219129 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.219016    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bkgx\" (UniqueName: \"kubernetes.io/projected/eaa6228d-8835-49c5-a0ed-77b810d4890f-kube-api-access-6bkgx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s\" (UID: \"eaa6228d-8835-49c5-a0ed-77b810d4890f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"
Jul 14 01:09:47.219129 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.219052    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eaa6228d-8835-49c5-a0ed-77b810d4890f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s\" (UID: \"eaa6228d-8835-49c5-a0ed-77b810d4890f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"
Jul 14 01:09:47.314062 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.314029    2567 generic.go:356] "Generic (PLEG): container finished" podID="f519d49d-6e68-448b-b529-981875cdc8cc" containerID="58ba00e7ea241dca34ef4da9a040ad744999be5db016a20e9e6d60705f82b9bd" exitCode=0
Jul 14 01:09:47.314259 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.314115    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4" event={"ID":"f519d49d-6e68-448b-b529-981875cdc8cc","Type":"ContainerDied","Data":"58ba00e7ea241dca34ef4da9a040ad744999be5db016a20e9e6d60705f82b9bd"}
Jul 14 01:09:47.314259 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.314167    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4" event={"ID":"f519d49d-6e68-448b-b529-981875cdc8cc","Type":"ContainerStarted","Data":"79f72310d267737c304c3ec04cb8003131a89b61b56c301f05ddbfd70d155987"}
Jul 14 01:09:47.314678 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.314662    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 01:09:47.315867 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.315839    2567 generic.go:356] "Generic (PLEG): container finished" podID="fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86" containerID="1cb04a261b223bef4c4a4f86e08c7532dc562182d3365f2c9ad97e769adec537" exitCode=0
Jul 14 01:09:47.315983 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.315880    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt" event={"ID":"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86","Type":"ContainerDied","Data":"1cb04a261b223bef4c4a4f86e08c7532dc562182d3365f2c9ad97e769adec537"}
Jul 14 01:09:47.315983 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.315916    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt" event={"ID":"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86","Type":"ContainerStarted","Data":"c84c28069ee0f1785a6a86ba8042ab2df6f8227b278d87c980b9dd88a0507429"}
Jul 14 01:09:47.319673 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.319645    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eaa6228d-8835-49c5-a0ed-77b810d4890f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s\" (UID: \"eaa6228d-8835-49c5-a0ed-77b810d4890f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"
Jul 14 01:09:47.319799 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.319687    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bkgx\" (UniqueName: \"kubernetes.io/projected/eaa6228d-8835-49c5-a0ed-77b810d4890f-kube-api-access-6bkgx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s\" (UID: \"eaa6228d-8835-49c5-a0ed-77b810d4890f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"
Jul 14 01:09:47.319799 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.319736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eaa6228d-8835-49c5-a0ed-77b810d4890f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s\" (UID: \"eaa6228d-8835-49c5-a0ed-77b810d4890f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"
Jul 14 01:09:47.320080 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.320059    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eaa6228d-8835-49c5-a0ed-77b810d4890f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s\" (UID: \"eaa6228d-8835-49c5-a0ed-77b810d4890f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"
Jul 14 01:09:47.320145 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.320094    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eaa6228d-8835-49c5-a0ed-77b810d4890f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s\" (UID: \"eaa6228d-8835-49c5-a0ed-77b810d4890f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"
Jul 14 01:09:47.328415 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.328379    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bkgx\" (UniqueName: \"kubernetes.io/projected/eaa6228d-8835-49c5-a0ed-77b810d4890f-kube-api-access-6bkgx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s\" (UID: \"eaa6228d-8835-49c5-a0ed-77b810d4890f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"
Jul 14 01:09:47.440501 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.440463    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"
Jul 14 01:09:47.512695 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.512661    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"]
Jul 14 01:09:47.515202 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.515183    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"
Jul 14 01:09:47.524855 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.524751    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"]
Jul 14 01:09:47.579477 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.579444    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"]
Jul 14 01:09:47.581917 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:09:47.581893    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa6228d_8835_49c5_a0ed_77b810d4890f.slice/crio-c46d1e86b4f9f070d2b8dca7c66bd9b033c0b228e7bc8f4b3909ec63ccbb93a6 WatchSource:0}: Error finding container c46d1e86b4f9f070d2b8dca7c66bd9b033c0b228e7bc8f4b3909ec63ccbb93a6: Status 404 returned error can't find the container with id c46d1e86b4f9f070d2b8dca7c66bd9b033c0b228e7bc8f4b3909ec63ccbb93a6
Jul 14 01:09:47.621990 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.621936    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7149f3c0-165a-4865-acaa-5e84b34bfdd3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88\" (UID: \"7149f3c0-165a-4865-acaa-5e84b34bfdd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"
Jul 14 01:09:47.622137 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.621998    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk9x5\" (UniqueName: \"kubernetes.io/projected/7149f3c0-165a-4865-acaa-5e84b34bfdd3-kube-api-access-xk9x5\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88\" (UID: \"7149f3c0-165a-4865-acaa-5e84b34bfdd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"
Jul 14 01:09:47.622137 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.622039    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7149f3c0-165a-4865-acaa-5e84b34bfdd3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88\" (UID: \"7149f3c0-165a-4865-acaa-5e84b34bfdd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"
Jul 14 01:09:47.723423 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.723384    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7149f3c0-165a-4865-acaa-5e84b34bfdd3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88\" (UID: \"7149f3c0-165a-4865-acaa-5e84b34bfdd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"
Jul 14 01:09:47.723423 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.723422    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk9x5\" (UniqueName: \"kubernetes.io/projected/7149f3c0-165a-4865-acaa-5e84b34bfdd3-kube-api-access-xk9x5\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88\" (UID: \"7149f3c0-165a-4865-acaa-5e84b34bfdd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"
Jul 14 01:09:47.723676 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.723617    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7149f3c0-165a-4865-acaa-5e84b34bfdd3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88\" (UID: \"7149f3c0-165a-4865-acaa-5e84b34bfdd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"
Jul 14 01:09:47.723840 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.723817    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7149f3c0-165a-4865-acaa-5e84b34bfdd3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88\" (UID: \"7149f3c0-165a-4865-acaa-5e84b34bfdd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"
Jul 14 01:09:47.723916 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.723901    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7149f3c0-165a-4865-acaa-5e84b34bfdd3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88\" (UID: \"7149f3c0-165a-4865-acaa-5e84b34bfdd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"
Jul 14 01:09:47.731612 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.731586    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk9x5\" (UniqueName: \"kubernetes.io/projected/7149f3c0-165a-4865-acaa-5e84b34bfdd3-kube-api-access-xk9x5\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88\" (UID: \"7149f3c0-165a-4865-acaa-5e84b34bfdd3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"
Jul 14 01:09:47.812896 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.812801    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:09:47.826597 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.826546    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"
Jul 14 01:09:47.970304 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:47.970270    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"]
Jul 14 01:09:48.001922 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:09:48.001880    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7149f3c0_165a_4865_acaa_5e84b34bfdd3.slice/crio-eb9f4787b8ea59858eb23de9432aae3d5423e2dad822e329a990e2b85c00057f WatchSource:0}: Error finding container eb9f4787b8ea59858eb23de9432aae3d5423e2dad822e329a990e2b85c00057f: Status 404 returned error can't find the container with id eb9f4787b8ea59858eb23de9432aae3d5423e2dad822e329a990e2b85c00057f
Jul 14 01:09:48.324627 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:48.324592    2567 generic.go:356] "Generic (PLEG): container finished" podID="eaa6228d-8835-49c5-a0ed-77b810d4890f" containerID="e61b6ee8b6e07c3b86986f5952ef0c17069c3b3e08dd18de87bb09fb4b6686ec" exitCode=0
Jul 14 01:09:48.324802 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:48.324671    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s" event={"ID":"eaa6228d-8835-49c5-a0ed-77b810d4890f","Type":"ContainerDied","Data":"e61b6ee8b6e07c3b86986f5952ef0c17069c3b3e08dd18de87bb09fb4b6686ec"}
Jul 14 01:09:48.324802 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:48.324719    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s" event={"ID":"eaa6228d-8835-49c5-a0ed-77b810d4890f","Type":"ContainerStarted","Data":"c46d1e86b4f9f070d2b8dca7c66bd9b033c0b228e7bc8f4b3909ec63ccbb93a6"}
Jul 14 01:09:48.326506 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:48.326482    2567 generic.go:356] "Generic (PLEG): container finished" podID="fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86" containerID="aa3e438f1af3b5f49322697e7cfa34ab00cb39b8db45c8972f6813ec6e9fe965" exitCode=0
Jul 14 01:09:48.326629 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:48.326569    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt" event={"ID":"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86","Type":"ContainerDied","Data":"aa3e438f1af3b5f49322697e7cfa34ab00cb39b8db45c8972f6813ec6e9fe965"}
Jul 14 01:09:48.328618 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:48.328585    2567 generic.go:356] "Generic (PLEG): container finished" podID="7149f3c0-165a-4865-acaa-5e84b34bfdd3" containerID="cd54f867b097f3595890f9331d4d6c5db17f4d49d129a94795f0685046da9ece" exitCode=0
Jul 14 01:09:48.328769 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:48.328658    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88" event={"ID":"7149f3c0-165a-4865-acaa-5e84b34bfdd3","Type":"ContainerDied","Data":"cd54f867b097f3595890f9331d4d6c5db17f4d49d129a94795f0685046da9ece"}
Jul 14 01:09:48.328769 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:48.328698    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88" event={"ID":"7149f3c0-165a-4865-acaa-5e84b34bfdd3","Type":"ContainerStarted","Data":"eb9f4787b8ea59858eb23de9432aae3d5423e2dad822e329a990e2b85c00057f"}
Jul 14 01:09:49.032843 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:49.032811    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:09:49.336207 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:49.336173    2567 generic.go:356] "Generic (PLEG): container finished" podID="fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86" containerID="7897bc45cb34d8dd83ee1f7712151e1deb65705c6d406218b7a50aa895ef4614" exitCode=0
Jul 14 01:09:49.336392 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:49.336255    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt" event={"ID":"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86","Type":"ContainerDied","Data":"7897bc45cb34d8dd83ee1f7712151e1deb65705c6d406218b7a50aa895ef4614"}
Jul 14 01:09:49.337995 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:49.337942    2567 generic.go:356] "Generic (PLEG): container finished" podID="f519d49d-6e68-448b-b529-981875cdc8cc" containerID="67faf4bccf8e99ffced6ae2b6ba0a864df309ae29eb2d26e3c2a1a3ca710ada9" exitCode=0
Jul 14 01:09:49.338140 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:49.337995    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4" event={"ID":"f519d49d-6e68-448b-b529-981875cdc8cc","Type":"ContainerDied","Data":"67faf4bccf8e99ffced6ae2b6ba0a864df309ae29eb2d26e3c2a1a3ca710ada9"}
Jul 14 01:09:49.340532 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:49.340501    2567 generic.go:356] "Generic (PLEG): container finished" podID="7149f3c0-165a-4865-acaa-5e84b34bfdd3" containerID="54a91baddd517b3ce2376137fb02ccb5ade6d3cbb01d59f0b5408d2dd972a766" exitCode=0
Jul 14 01:09:49.340620 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:49.340584    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88" event={"ID":"7149f3c0-165a-4865-acaa-5e84b34bfdd3","Type":"ContainerDied","Data":"54a91baddd517b3ce2376137fb02ccb5ade6d3cbb01d59f0b5408d2dd972a766"}
Jul 14 01:09:49.342699 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:49.342672    2567 generic.go:356] "Generic (PLEG): container finished" podID="eaa6228d-8835-49c5-a0ed-77b810d4890f" containerID="eea17877c3d97126176392cc7eb7bde68db9e870b67041aab4e6a56e251aa07a" exitCode=0
Jul 14 01:09:49.342799 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:49.342715    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s" event={"ID":"eaa6228d-8835-49c5-a0ed-77b810d4890f","Type":"ContainerDied","Data":"eea17877c3d97126176392cc7eb7bde68db9e870b67041aab4e6a56e251aa07a"}
Jul 14 01:09:50.350802 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.350771    2567 generic.go:356] "Generic (PLEG): container finished" podID="f519d49d-6e68-448b-b529-981875cdc8cc" containerID="b92c2d7e877fc140dbe270018048239dc905c8f98ae0954bf27d43a1ece70dc8" exitCode=0
Jul 14 01:09:50.351262 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.350850    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4" event={"ID":"f519d49d-6e68-448b-b529-981875cdc8cc","Type":"ContainerDied","Data":"b92c2d7e877fc140dbe270018048239dc905c8f98ae0954bf27d43a1ece70dc8"}
Jul 14 01:09:50.352825 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.352803    2567 generic.go:356] "Generic (PLEG): container finished" podID="7149f3c0-165a-4865-acaa-5e84b34bfdd3" containerID="9123f269e3b092ddf4b4d19c30a07a448ed08b783e86a698fff82fe7badd00b8" exitCode=0
Jul 14 01:09:50.352928 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.352880    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88" event={"ID":"7149f3c0-165a-4865-acaa-5e84b34bfdd3","Type":"ContainerDied","Data":"9123f269e3b092ddf4b4d19c30a07a448ed08b783e86a698fff82fe7badd00b8"}
Jul 14 01:09:50.354798 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.354774    2567 generic.go:356] "Generic (PLEG): container finished" podID="eaa6228d-8835-49c5-a0ed-77b810d4890f" containerID="cd778bee227691413a2127bb9f79396c9dfe88f759a1a91276d5cddd7f16148c" exitCode=0
Jul 14 01:09:50.354905 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.354846    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s" event={"ID":"eaa6228d-8835-49c5-a0ed-77b810d4890f","Type":"ContainerDied","Data":"cd778bee227691413a2127bb9f79396c9dfe88f759a1a91276d5cddd7f16148c"}
Jul 14 01:09:50.485775 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.485746    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"
Jul 14 01:09:50.646909 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.646866    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b94t\" (UniqueName: \"kubernetes.io/projected/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-kube-api-access-8b94t\") pod \"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86\" (UID: \"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86\") "
Jul 14 01:09:50.647109 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.647003    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-bundle\") pod \"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86\" (UID: \"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86\") "
Jul 14 01:09:50.647158 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.647112    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-util\") pod \"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86\" (UID: \"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86\") "
Jul 14 01:09:50.647520 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.647499    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-bundle" (OuterVolumeSpecName: "bundle") pod "fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86" (UID: "fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:09:50.649391 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.649364    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-kube-api-access-8b94t" (OuterVolumeSpecName: "kube-api-access-8b94t") pod "fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86" (UID: "fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86"). InnerVolumeSpecName "kube-api-access-8b94t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:09:50.652883 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.652861    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-util" (OuterVolumeSpecName: "util") pod "fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86" (UID: "fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:09:50.748623 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.748535    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:09:50.748623 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.748567    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-util\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:09:50.748623 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:50.748577    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b94t\" (UniqueName: \"kubernetes.io/projected/fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86-kube-api-access-8b94t\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:09:51.361696 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.361659    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt"
Jul 14 01:09:51.361696 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.361689    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt" event={"ID":"fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86","Type":"ContainerDied","Data":"c84c28069ee0f1785a6a86ba8042ab2df6f8227b278d87c980b9dd88a0507429"}
Jul 14 01:09:51.362236 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.361739    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c84c28069ee0f1785a6a86ba8042ab2df6f8227b278d87c980b9dd88a0507429"
Jul 14 01:09:51.507181 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.507146    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"
Jul 14 01:09:51.555672 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.555645    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"
Jul 14 01:09:51.558877 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.558846    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"
Jul 14 01:09:51.655268 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.655179    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wm5p\" (UniqueName: \"kubernetes.io/projected/f519d49d-6e68-448b-b529-981875cdc8cc-kube-api-access-6wm5p\") pod \"f519d49d-6e68-448b-b529-981875cdc8cc\" (UID: \"f519d49d-6e68-448b-b529-981875cdc8cc\") "
Jul 14 01:09:51.655268 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.655212    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bkgx\" (UniqueName: \"kubernetes.io/projected/eaa6228d-8835-49c5-a0ed-77b810d4890f-kube-api-access-6bkgx\") pod \"eaa6228d-8835-49c5-a0ed-77b810d4890f\" (UID: \"eaa6228d-8835-49c5-a0ed-77b810d4890f\") "
Jul 14 01:09:51.655268 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.655251    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f519d49d-6e68-448b-b529-981875cdc8cc-util\") pod \"f519d49d-6e68-448b-b529-981875cdc8cc\" (UID: \"f519d49d-6e68-448b-b529-981875cdc8cc\") "
Jul 14 01:09:51.655548 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.655286    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk9x5\" (UniqueName: \"kubernetes.io/projected/7149f3c0-165a-4865-acaa-5e84b34bfdd3-kube-api-access-xk9x5\") pod \"7149f3c0-165a-4865-acaa-5e84b34bfdd3\" (UID: \"7149f3c0-165a-4865-acaa-5e84b34bfdd3\") "
Jul 14 01:09:51.655548 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.655335    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7149f3c0-165a-4865-acaa-5e84b34bfdd3-util\") pod \"7149f3c0-165a-4865-acaa-5e84b34bfdd3\" (UID: \"7149f3c0-165a-4865-acaa-5e84b34bfdd3\") "
Jul 14 01:09:51.655548 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.655392    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eaa6228d-8835-49c5-a0ed-77b810d4890f-util\") pod \"eaa6228d-8835-49c5-a0ed-77b810d4890f\" (UID: \"eaa6228d-8835-49c5-a0ed-77b810d4890f\") "
Jul 14 01:09:51.655548 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.655433    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7149f3c0-165a-4865-acaa-5e84b34bfdd3-bundle\") pod \"7149f3c0-165a-4865-acaa-5e84b34bfdd3\" (UID: \"7149f3c0-165a-4865-acaa-5e84b34bfdd3\") "
Jul 14 01:09:51.655548 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.655457    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f519d49d-6e68-448b-b529-981875cdc8cc-bundle\") pod \"f519d49d-6e68-448b-b529-981875cdc8cc\" (UID: \"f519d49d-6e68-448b-b529-981875cdc8cc\") "
Jul 14 01:09:51.655548 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.655477    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eaa6228d-8835-49c5-a0ed-77b810d4890f-bundle\") pod \"eaa6228d-8835-49c5-a0ed-77b810d4890f\" (UID: \"eaa6228d-8835-49c5-a0ed-77b810d4890f\") "
Jul 14 01:09:51.656545 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.656483    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7149f3c0-165a-4865-acaa-5e84b34bfdd3-bundle" (OuterVolumeSpecName: "bundle") pod "7149f3c0-165a-4865-acaa-5e84b34bfdd3" (UID: "7149f3c0-165a-4865-acaa-5e84b34bfdd3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:09:51.656777 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.656750    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7149f3c0-165a-4865-acaa-5e84b34bfdd3-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:09:51.657456 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.657418    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f519d49d-6e68-448b-b529-981875cdc8cc-bundle" (OuterVolumeSpecName: "bundle") pod "f519d49d-6e68-448b-b529-981875cdc8cc" (UID: "f519d49d-6e68-448b-b529-981875cdc8cc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:09:51.657879 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.657840    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa6228d-8835-49c5-a0ed-77b810d4890f-bundle" (OuterVolumeSpecName: "bundle") pod "eaa6228d-8835-49c5-a0ed-77b810d4890f" (UID: "eaa6228d-8835-49c5-a0ed-77b810d4890f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:09:51.658051 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.658030    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7149f3c0-165a-4865-acaa-5e84b34bfdd3-kube-api-access-xk9x5" (OuterVolumeSpecName: "kube-api-access-xk9x5") pod "7149f3c0-165a-4865-acaa-5e84b34bfdd3" (UID: "7149f3c0-165a-4865-acaa-5e84b34bfdd3"). InnerVolumeSpecName "kube-api-access-xk9x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:09:51.658140 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.658061    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f519d49d-6e68-448b-b529-981875cdc8cc-kube-api-access-6wm5p" (OuterVolumeSpecName: "kube-api-access-6wm5p") pod "f519d49d-6e68-448b-b529-981875cdc8cc" (UID: "f519d49d-6e68-448b-b529-981875cdc8cc"). InnerVolumeSpecName "kube-api-access-6wm5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:09:51.658352 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.658327    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa6228d-8835-49c5-a0ed-77b810d4890f-kube-api-access-6bkgx" (OuterVolumeSpecName: "kube-api-access-6bkgx") pod "eaa6228d-8835-49c5-a0ed-77b810d4890f" (UID: "eaa6228d-8835-49c5-a0ed-77b810d4890f"). InnerVolumeSpecName "kube-api-access-6bkgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:09:51.662226 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.662204    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f519d49d-6e68-448b-b529-981875cdc8cc-util" (OuterVolumeSpecName: "util") pod "f519d49d-6e68-448b-b529-981875cdc8cc" (UID: "f519d49d-6e68-448b-b529-981875cdc8cc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:09:51.662733 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.662711    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa6228d-8835-49c5-a0ed-77b810d4890f-util" (OuterVolumeSpecName: "util") pod "eaa6228d-8835-49c5-a0ed-77b810d4890f" (UID: "eaa6228d-8835-49c5-a0ed-77b810d4890f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:09:51.664561 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.664534    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7149f3c0-165a-4865-acaa-5e84b34bfdd3-util" (OuterVolumeSpecName: "util") pod "7149f3c0-165a-4865-acaa-5e84b34bfdd3" (UID: "7149f3c0-165a-4865-acaa-5e84b34bfdd3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:09:51.757449 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.757413    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f519d49d-6e68-448b-b529-981875cdc8cc-util\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:09:51.757449 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.757446    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk9x5\" (UniqueName: \"kubernetes.io/projected/7149f3c0-165a-4865-acaa-5e84b34bfdd3-kube-api-access-xk9x5\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:09:51.757449 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.757456    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7149f3c0-165a-4865-acaa-5e84b34bfdd3-util\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:09:51.757449 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.757464    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eaa6228d-8835-49c5-a0ed-77b810d4890f-util\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:09:51.757706 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.757472    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f519d49d-6e68-448b-b529-981875cdc8cc-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:09:51.757706 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.757481    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eaa6228d-8835-49c5-a0ed-77b810d4890f-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:09:51.757706 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.757489    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wm5p\" (UniqueName: \"kubernetes.io/projected/f519d49d-6e68-448b-b529-981875cdc8cc-kube-api-access-6wm5p\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:09:51.757706 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:51.757497    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bkgx\" (UniqueName: \"kubernetes.io/projected/eaa6228d-8835-49c5-a0ed-77b810d4890f-kube-api-access-6bkgx\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:09:52.368164 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:52.368136    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4"
Jul 14 01:09:52.368164 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:52.368150    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4" event={"ID":"f519d49d-6e68-448b-b529-981875cdc8cc","Type":"ContainerDied","Data":"79f72310d267737c304c3ec04cb8003131a89b61b56c301f05ddbfd70d155987"}
Jul 14 01:09:52.368690 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:52.368205    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79f72310d267737c304c3ec04cb8003131a89b61b56c301f05ddbfd70d155987"
Jul 14 01:09:52.370258 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:52.370235    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88"
Jul 14 01:09:52.370407 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:52.370262    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88" event={"ID":"7149f3c0-165a-4865-acaa-5e84b34bfdd3","Type":"ContainerDied","Data":"eb9f4787b8ea59858eb23de9432aae3d5423e2dad822e329a990e2b85c00057f"}
Jul 14 01:09:52.370407 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:52.370304    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb9f4787b8ea59858eb23de9432aae3d5423e2dad822e329a990e2b85c00057f"
Jul 14 01:09:52.372588 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:52.372567    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s"
Jul 14 01:09:52.372588 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:52.372583    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s" event={"ID":"eaa6228d-8835-49c5-a0ed-77b810d4890f","Type":"ContainerDied","Data":"c46d1e86b4f9f070d2b8dca7c66bd9b033c0b228e7bc8f4b3909ec63ccbb93a6"}
Jul 14 01:09:52.372779 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:52.372624    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c46d1e86b4f9f070d2b8dca7c66bd9b033c0b228e7bc8f4b3909ec63ccbb93a6"
Jul 14 01:09:52.391518 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:09:52.391486    2567 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf519d49d_6e68_448b_b529_981875cdc8cc.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7149f3c0_165a_4865_acaa_5e84b34bfdd3.slice\": RecentStats: unable to find data in memory cache]"
Jul 14 01:09:52.723819 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:52.723720    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:09:52.781271 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:52.781235    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:09:53.566076 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:53.566036    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:09:57.209584 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:57.209546    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:09:59.112646 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:59.112605    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:09:59.213087 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:59.213043    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:09:59.378223 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:09:59.378189    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:01.182052 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:01.182014    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:01.409128 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:01.409099    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:01.577355 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:01.577264    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:04.375119 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:04.375081    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:04.475003 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:04.474951    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:04.794880 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:04.794794    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:05.029973 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:05.029931    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:06.179791 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:06.179755    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:06.287884 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:06.287840    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:08.032611 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:08.032579    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:08.126560 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:08.126526    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:08.440555 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:08.440526    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:09.234107 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:09.234068    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:09.320321 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:09.320287    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:09.457987 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:09.457940    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-867df57d85-mt7qf"]
Jul 14 01:10:09.558021 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:09.557909    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:09.660368 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:09.660337    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:09.768558 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:09.768528    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:09.869250 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:09.869220    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:17.071084 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:17.071052    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:17.213868 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:17.213839    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:22.814648 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:22.814617    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:22.875979 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:22.875930    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:25.788247 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:25.788212    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:25.889170 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:25.889141    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:29.596670 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:29.596636    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:29.699937 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:29.699900    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:34.486929 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.486887    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-867df57d85-mt7qf" podUID="e7c58e8d-5d60-401b-aabc-b83fbe67712c" containerName="console" containerID="cri-o://92d03d7c09157ff24229d6520491c21f65880d13d8213a8a562a8c29ff33a64f" gracePeriod=15
Jul 14 01:10:34.745278 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.745208    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:10:34.821854 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.821819    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-config\") pod \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") "
Jul 14 01:10:34.822073 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.821909    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-oauth-config\") pod \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") "
Jul 14 01:10:34.822073 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.821930    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-oauth-serving-cert\") pod \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") "
Jul 14 01:10:34.822073 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.821992    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-serving-cert\") pod \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") "
Jul 14 01:10:34.822073 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.822028    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-trusted-ca-bundle\") pod \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") "
Jul 14 01:10:34.822073 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.822053    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6m86\" (UniqueName: \"kubernetes.io/projected/e7c58e8d-5d60-401b-aabc-b83fbe67712c-kube-api-access-p6m86\") pod \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") "
Jul 14 01:10:34.822449 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.822089    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-service-ca\") pod \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\" (UID: \"e7c58e8d-5d60-401b-aabc-b83fbe67712c\") "
Jul 14 01:10:34.822449 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.822392    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e7c58e8d-5d60-401b-aabc-b83fbe67712c" (UID: "e7c58e8d-5d60-401b-aabc-b83fbe67712c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:10:34.822679 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.822657    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-service-ca" (OuterVolumeSpecName: "service-ca") pod "e7c58e8d-5d60-401b-aabc-b83fbe67712c" (UID: "e7c58e8d-5d60-401b-aabc-b83fbe67712c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:10:34.822745 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.822718    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-config" (OuterVolumeSpecName: "console-config") pod "e7c58e8d-5d60-401b-aabc-b83fbe67712c" (UID: "e7c58e8d-5d60-401b-aabc-b83fbe67712c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:10:34.822869 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.822733    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e7c58e8d-5d60-401b-aabc-b83fbe67712c" (UID: "e7c58e8d-5d60-401b-aabc-b83fbe67712c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:10:34.824345 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.824315    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e7c58e8d-5d60-401b-aabc-b83fbe67712c" (UID: "e7c58e8d-5d60-401b-aabc-b83fbe67712c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:10:34.824528 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.824502    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e7c58e8d-5d60-401b-aabc-b83fbe67712c" (UID: "e7c58e8d-5d60-401b-aabc-b83fbe67712c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:10:34.824581 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.824565    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c58e8d-5d60-401b-aabc-b83fbe67712c-kube-api-access-p6m86" (OuterVolumeSpecName: "kube-api-access-p6m86") pod "e7c58e8d-5d60-401b-aabc-b83fbe67712c" (UID: "e7c58e8d-5d60-401b-aabc-b83fbe67712c"). InnerVolumeSpecName "kube-api-access-p6m86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:10:34.923579 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.923536    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-oauth-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:10:34.923579 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.923568    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-oauth-serving-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:10:34.923579 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.923580    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-serving-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:10:34.923579 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.923588    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-trusted-ca-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:10:34.923855 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.923597    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6m86\" (UniqueName: \"kubernetes.io/projected/e7c58e8d-5d60-401b-aabc-b83fbe67712c-kube-api-access-p6m86\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:10:34.923855 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.923606    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:10:34.923855 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:34.923614    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7c58e8d-5d60-401b-aabc-b83fbe67712c-console-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:10:35.586177 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:35.586141    2567 generic.go:356] "Generic (PLEG): container finished" podID="e7c58e8d-5d60-401b-aabc-b83fbe67712c" containerID="92d03d7c09157ff24229d6520491c21f65880d13d8213a8a562a8c29ff33a64f" exitCode=0
Jul 14 01:10:35.586651 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:35.586183    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-867df57d85-mt7qf"
Jul 14 01:10:35.586651 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:35.586231    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-867df57d85-mt7qf" event={"ID":"e7c58e8d-5d60-401b-aabc-b83fbe67712c","Type":"ContainerDied","Data":"92d03d7c09157ff24229d6520491c21f65880d13d8213a8a562a8c29ff33a64f"}
Jul 14 01:10:35.586651 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:35.586287    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-867df57d85-mt7qf" event={"ID":"e7c58e8d-5d60-401b-aabc-b83fbe67712c","Type":"ContainerDied","Data":"e5e379a5eb78a2376b2ad5dc16600ba46bec3c9846a6b8fc856e88951ad7f857"}
Jul 14 01:10:35.586651 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:35.586308    2567 scope.go:117] "RemoveContainer" containerID="92d03d7c09157ff24229d6520491c21f65880d13d8213a8a562a8c29ff33a64f"
Jul 14 01:10:35.595302 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:35.595278    2567 scope.go:117] "RemoveContainer" containerID="92d03d7c09157ff24229d6520491c21f65880d13d8213a8a562a8c29ff33a64f"
Jul 14 01:10:35.595614 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:10:35.595592    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d03d7c09157ff24229d6520491c21f65880d13d8213a8a562a8c29ff33a64f\": container with ID starting with 92d03d7c09157ff24229d6520491c21f65880d13d8213a8a562a8c29ff33a64f not found: ID does not exist" containerID="92d03d7c09157ff24229d6520491c21f65880d13d8213a8a562a8c29ff33a64f"
Jul 14 01:10:35.595659 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:35.595624    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d03d7c09157ff24229d6520491c21f65880d13d8213a8a562a8c29ff33a64f"} err="failed to get container status \"92d03d7c09157ff24229d6520491c21f65880d13d8213a8a562a8c29ff33a64f\": rpc error: code = NotFound desc = could not find container \"92d03d7c09157ff24229d6520491c21f65880d13d8213a8a562a8c29ff33a64f\": container with ID starting with 92d03d7c09157ff24229d6520491c21f65880d13d8213a8a562a8c29ff33a64f not found: ID does not exist"
Jul 14 01:10:35.600650 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:35.600621    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-867df57d85-mt7qf"]
Jul 14 01:10:35.603712 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:35.603686    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-867df57d85-mt7qf"]
Jul 14 01:10:35.892847 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:35.892816    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:35.940352 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:35.940322    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:36.974394 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:36.974363    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7c58e8d-5d60-401b-aabc-b83fbe67712c" path="/var/lib/kubelet/pods/e7c58e8d-5d60-401b-aabc-b83fbe67712c/volumes"
Jul 14 01:10:42.447994 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:42.447946    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:42.547476 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:42.547436    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:45.444266 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:45.444232    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:45.561783 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:45.561744    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:47.927164 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:47.927128    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:48.220883 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:48.220777    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:49.449192 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:49.449159    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:49.513893 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:49.513863    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:52.743060 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:52.743028    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:52.845189 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:52.845148    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:52.944976 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:52.944928    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:53.545944 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:53.545905    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:53.644091 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:53.644044    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:57.444794 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:57.444762    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:10:57.545507 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:10:57.545470    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:03.643584 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:03.643534    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:03.696976 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:03.696925    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:03.844721 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:03.844684    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:03.910557 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:03.910455    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:06.056241 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:06.056205    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:06.156356 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:06.156320    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:06.881562 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:06.881516    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:13.995672 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:13.995638    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:14.096200 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:14.096163    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:14.543655 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:14.543623    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:27.587669 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:27.587584    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:27.707575 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:27.707541    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:35.707439 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.707393    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:35.720509 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720473    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d65bf78b5-tf5zs"]
Jul 14 01:11:35.720796 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720784    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86" containerName="extract"
Jul 14 01:11:35.720865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720797    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86" containerName="extract"
Jul 14 01:11:35.720865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720808    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c58e8d-5d60-401b-aabc-b83fbe67712c" containerName="console"
Jul 14 01:11:35.720865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720813    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c58e8d-5d60-401b-aabc-b83fbe67712c" containerName="console"
Jul 14 01:11:35.720865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720822    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f519d49d-6e68-448b-b529-981875cdc8cc" containerName="extract"
Jul 14 01:11:35.720865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720828    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f519d49d-6e68-448b-b529-981875cdc8cc" containerName="extract"
Jul 14 01:11:35.720865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720836    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86" containerName="pull"
Jul 14 01:11:35.720865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720841    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86" containerName="pull"
Jul 14 01:11:35.720865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720847    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7149f3c0-165a-4865-acaa-5e84b34bfdd3" containerName="extract"
Jul 14 01:11:35.720865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720852    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7149f3c0-165a-4865-acaa-5e84b34bfdd3" containerName="extract"
Jul 14 01:11:35.720865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720861    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7149f3c0-165a-4865-acaa-5e84b34bfdd3" containerName="util"
Jul 14 01:11:35.720865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720866    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7149f3c0-165a-4865-acaa-5e84b34bfdd3" containerName="util"
Jul 14 01:11:35.720865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720872    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f519d49d-6e68-448b-b529-981875cdc8cc" containerName="util"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720879    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f519d49d-6e68-448b-b529-981875cdc8cc" containerName="util"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720887    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaa6228d-8835-49c5-a0ed-77b810d4890f" containerName="util"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720895    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaa6228d-8835-49c5-a0ed-77b810d4890f" containerName="util"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720905    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaa6228d-8835-49c5-a0ed-77b810d4890f" containerName="extract"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720911    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaa6228d-8835-49c5-a0ed-77b810d4890f" containerName="extract"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720918    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f519d49d-6e68-448b-b529-981875cdc8cc" containerName="pull"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720923    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f519d49d-6e68-448b-b529-981875cdc8cc" containerName="pull"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720928    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaa6228d-8835-49c5-a0ed-77b810d4890f" containerName="pull"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720933    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaa6228d-8835-49c5-a0ed-77b810d4890f" containerName="pull"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720938    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7149f3c0-165a-4865-acaa-5e84b34bfdd3" containerName="pull"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720942    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7149f3c0-165a-4865-acaa-5e84b34bfdd3" containerName="pull"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720950    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86" containerName="util"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.720973    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86" containerName="util"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.721060    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86" containerName="extract"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.721072    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaa6228d-8835-49c5-a0ed-77b810d4890f" containerName="extract"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.721080    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7149f3c0-165a-4865-acaa-5e84b34bfdd3" containerName="extract"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.721088    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c58e8d-5d60-401b-aabc-b83fbe67712c" containerName="console"
Jul 14 01:11:35.721318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.721096    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f519d49d-6e68-448b-b529-981875cdc8cc" containerName="extract"
Jul 14 01:11:35.725259 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.725235    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d65bf78b5-tf5zs"
Jul 14 01:11:35.727267 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.727242    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 14 01:11:35.727463 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.727270    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 14 01:11:35.727525 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.727512    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zrmp8\""
Jul 14 01:11:35.735586 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.735556    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d65bf78b5-tf5zs"]
Jul 14 01:11:35.797162 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.797133    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:35.830148 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.830103    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9678d375-9dfd-442d-a9f6-5903b4b616d8-webhook-cert\") pod \"maas-controller-5d65bf78b5-tf5zs\" (UID: \"9678d375-9dfd-442d-a9f6-5903b4b616d8\") " pod="opendatahub/maas-controller-5d65bf78b5-tf5zs"
Jul 14 01:11:35.830148 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.830160    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9224z\" (UniqueName: \"kubernetes.io/projected/9678d375-9dfd-442d-a9f6-5903b4b616d8-kube-api-access-9224z\") pod \"maas-controller-5d65bf78b5-tf5zs\" (UID: \"9678d375-9dfd-442d-a9f6-5903b4b616d8\") " pod="opendatahub/maas-controller-5d65bf78b5-tf5zs"
Jul 14 01:11:35.879034 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.878996    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:35.930697 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.930670    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9224z\" (UniqueName: \"kubernetes.io/projected/9678d375-9dfd-442d-a9f6-5903b4b616d8-kube-api-access-9224z\") pod \"maas-controller-5d65bf78b5-tf5zs\" (UID: \"9678d375-9dfd-442d-a9f6-5903b4b616d8\") " pod="opendatahub/maas-controller-5d65bf78b5-tf5zs"
Jul 14 01:11:35.930889 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.930731    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9678d375-9dfd-442d-a9f6-5903b4b616d8-webhook-cert\") pod \"maas-controller-5d65bf78b5-tf5zs\" (UID: \"9678d375-9dfd-442d-a9f6-5903b4b616d8\") " pod="opendatahub/maas-controller-5d65bf78b5-tf5zs"
Jul 14 01:11:35.933456 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.933427    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9678d375-9dfd-442d-a9f6-5903b4b616d8-webhook-cert\") pod \"maas-controller-5d65bf78b5-tf5zs\" (UID: \"9678d375-9dfd-442d-a9f6-5903b4b616d8\") " pod="opendatahub/maas-controller-5d65bf78b5-tf5zs"
Jul 14 01:11:35.938412 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.938387    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9224z\" (UniqueName: \"kubernetes.io/projected/9678d375-9dfd-442d-a9f6-5903b4b616d8-kube-api-access-9224z\") pod \"maas-controller-5d65bf78b5-tf5zs\" (UID: \"9678d375-9dfd-442d-a9f6-5903b4b616d8\") " pod="opendatahub/maas-controller-5d65bf78b5-tf5zs"
Jul 14 01:11:35.958748 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:35.958668    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:36.036044 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:36.036008    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d65bf78b5-tf5zs"
Jul 14 01:11:36.177121 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:36.177097    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d65bf78b5-tf5zs"]
Jul 14 01:11:36.179132 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:11:36.179095    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9678d375_9dfd_442d_a9f6_5903b4b616d8.slice/crio-822343cf439f7d5326d7e6e9316ef807fd93692f2dc1cda3c414243741298467 WatchSource:0}: Error finding container 822343cf439f7d5326d7e6e9316ef807fd93692f2dc1cda3c414243741298467: Status 404 returned error can't find the container with id 822343cf439f7d5326d7e6e9316ef807fd93692f2dc1cda3c414243741298467
Jul 14 01:11:36.887080 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:36.887045    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d65bf78b5-tf5zs" event={"ID":"9678d375-9dfd-442d-a9f6-5903b4b616d8","Type":"ContainerStarted","Data":"822343cf439f7d5326d7e6e9316ef807fd93692f2dc1cda3c414243741298467"}
Jul 14 01:11:39.139089 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:39.139058    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:39.273250 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:39.273159    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:11:39.904520 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:39.904482    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d65bf78b5-tf5zs" event={"ID":"9678d375-9dfd-442d-a9f6-5903b4b616d8","Type":"ContainerStarted","Data":"6b5adea6458570d12d0ba72e49d35646aa2f8581a99c6a5f9f3c1ac50435fa6e"}
Jul 14 01:11:39.904719 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:39.904576    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d65bf78b5-tf5zs"
Jul 14 01:11:39.921202 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:39.921149    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5d65bf78b5-tf5zs" podStartSLOduration=2.14486724 podStartE2EDuration="4.921131193s" totalImagesPullingTime="2.776263953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:11:39.918786977 +0000 UTC m=+729.483196031" watchObservedRunningTime="2026-07-14 01:11:39.921131193 +0000 UTC m=+729.485540248"
Jul 14 01:11:50.391135 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.391097    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d65bf78b5-tf5zs"]
Jul 14 01:11:50.391526 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.391308    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5d65bf78b5-tf5zs" podUID="9678d375-9dfd-442d-a9f6-5903b4b616d8" containerName="manager" containerID="cri-o://6b5adea6458570d12d0ba72e49d35646aa2f8581a99c6a5f9f3c1ac50435fa6e" gracePeriod=30
Jul 14 01:11:50.393317 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.393284    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d65bf78b5-tf5zs"
Jul 14 01:11:50.643847 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.643770    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d65bf78b5-tf5zs"
Jul 14 01:11:50.764486 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.764440    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9678d375-9dfd-442d-a9f6-5903b4b616d8-webhook-cert\") pod \"9678d375-9dfd-442d-a9f6-5903b4b616d8\" (UID: \"9678d375-9dfd-442d-a9f6-5903b4b616d8\") "
Jul 14 01:11:50.764486 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.764497    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9224z\" (UniqueName: \"kubernetes.io/projected/9678d375-9dfd-442d-a9f6-5903b4b616d8-kube-api-access-9224z\") pod \"9678d375-9dfd-442d-a9f6-5903b4b616d8\" (UID: \"9678d375-9dfd-442d-a9f6-5903b4b616d8\") "
Jul 14 01:11:50.766759 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.766721    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9678d375-9dfd-442d-a9f6-5903b4b616d8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9678d375-9dfd-442d-a9f6-5903b4b616d8" (UID: "9678d375-9dfd-442d-a9f6-5903b4b616d8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:11:50.766893 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.766789    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9678d375-9dfd-442d-a9f6-5903b4b616d8-kube-api-access-9224z" (OuterVolumeSpecName: "kube-api-access-9224z") pod "9678d375-9dfd-442d-a9f6-5903b4b616d8" (UID: "9678d375-9dfd-442d-a9f6-5903b4b616d8"). InnerVolumeSpecName "kube-api-access-9224z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:11:50.865110 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.865069    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9224z\" (UniqueName: \"kubernetes.io/projected/9678d375-9dfd-442d-a9f6-5903b4b616d8-kube-api-access-9224z\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:11:50.865110 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.865100    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9678d375-9dfd-442d-a9f6-5903b4b616d8-webhook-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:11:50.957396 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.957311    2567 generic.go:356] "Generic (PLEG): container finished" podID="9678d375-9dfd-442d-a9f6-5903b4b616d8" containerID="6b5adea6458570d12d0ba72e49d35646aa2f8581a99c6a5f9f3c1ac50435fa6e" exitCode=0
Jul 14 01:11:50.957396 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.957361    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d65bf78b5-tf5zs"
Jul 14 01:11:50.957396 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.957372    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d65bf78b5-tf5zs" event={"ID":"9678d375-9dfd-442d-a9f6-5903b4b616d8","Type":"ContainerDied","Data":"6b5adea6458570d12d0ba72e49d35646aa2f8581a99c6a5f9f3c1ac50435fa6e"}
Jul 14 01:11:50.957621 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.957413    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d65bf78b5-tf5zs" event={"ID":"9678d375-9dfd-442d-a9f6-5903b4b616d8","Type":"ContainerDied","Data":"822343cf439f7d5326d7e6e9316ef807fd93692f2dc1cda3c414243741298467"}
Jul 14 01:11:50.957621 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.957433    2567 scope.go:117] "RemoveContainer" containerID="6b5adea6458570d12d0ba72e49d35646aa2f8581a99c6a5f9f3c1ac50435fa6e"
Jul 14 01:11:50.967242 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.967208    2567 scope.go:117] "RemoveContainer" containerID="6b5adea6458570d12d0ba72e49d35646aa2f8581a99c6a5f9f3c1ac50435fa6e"
Jul 14 01:11:50.967567 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:11:50.967547    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5adea6458570d12d0ba72e49d35646aa2f8581a99c6a5f9f3c1ac50435fa6e\": container with ID starting with 6b5adea6458570d12d0ba72e49d35646aa2f8581a99c6a5f9f3c1ac50435fa6e not found: ID does not exist" containerID="6b5adea6458570d12d0ba72e49d35646aa2f8581a99c6a5f9f3c1ac50435fa6e"
Jul 14 01:11:50.967654 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.967575    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5adea6458570d12d0ba72e49d35646aa2f8581a99c6a5f9f3c1ac50435fa6e"} err="failed to get container status \"6b5adea6458570d12d0ba72e49d35646aa2f8581a99c6a5f9f3c1ac50435fa6e\": rpc error: code = NotFound desc = could not find container \"6b5adea6458570d12d0ba72e49d35646aa2f8581a99c6a5f9f3c1ac50435fa6e\": container with ID starting with 6b5adea6458570d12d0ba72e49d35646aa2f8581a99c6a5f9f3c1ac50435fa6e not found: ID does not exist"
Jul 14 01:11:50.977453 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.977419    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d65bf78b5-tf5zs"]
Jul 14 01:11:50.981727 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:50.981684    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5d65bf78b5-tf5zs"]
Jul 14 01:11:52.973608 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:11:52.973575    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9678d375-9dfd-442d-a9f6-5903b4b616d8" path="/var/lib/kubelet/pods/9678d375-9dfd-442d-a9f6-5903b4b616d8/volumes"
Jul 14 01:12:13.596685 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:13.596612    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:12:13.694392 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:13.694355    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:12:14.017098 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.017056    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:12:14.133824 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.133789    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:12:14.518497 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.518463    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:12:14.640866 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.640828    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:12:14.731160 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.731121    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q"]
Jul 14 01:12:14.731438 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.731427    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9678d375-9dfd-442d-a9f6-5903b4b616d8" containerName="manager"
Jul 14 01:12:14.731482 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.731440    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9678d375-9dfd-442d-a9f6-5903b4b616d8" containerName="manager"
Jul 14 01:12:14.731517 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.731504    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9678d375-9dfd-442d-a9f6-5903b4b616d8" containerName="manager"
Jul 14 01:12:14.736816 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.736795    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q"
Jul 14 01:12:14.738820 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.738798    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-wvnc9\""
Jul 14 01:12:14.738820 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.738814    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 14 01:12:14.742430 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.742403    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q"]
Jul 14 01:12:14.758452 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.758423    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6lp\" (UniqueName: \"kubernetes.io/projected/b6934939-13dd-40c8-ae10-62971cf22838-kube-api-access-tb6lp\") pod \"payload-pre-processing-74584bb9f8-hsj5q\" (UID: \"b6934939-13dd-40c8-ae10-62971cf22838\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q"
Jul 14 01:12:14.758621 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.758518    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6934939-13dd-40c8-ae10-62971cf22838-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-hsj5q\" (UID: \"b6934939-13dd-40c8-ae10-62971cf22838\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q"
Jul 14 01:12:14.859333 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.859245    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6934939-13dd-40c8-ae10-62971cf22838-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-hsj5q\" (UID: \"b6934939-13dd-40c8-ae10-62971cf22838\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q"
Jul 14 01:12:14.859333 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.859297    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6lp\" (UniqueName: \"kubernetes.io/projected/b6934939-13dd-40c8-ae10-62971cf22838-kube-api-access-tb6lp\") pod \"payload-pre-processing-74584bb9f8-hsj5q\" (UID: \"b6934939-13dd-40c8-ae10-62971cf22838\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q"
Jul 14 01:12:14.860029 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.860007    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6934939-13dd-40c8-ae10-62971cf22838-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-hsj5q\" (UID: \"b6934939-13dd-40c8-ae10-62971cf22838\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q"
Jul 14 01:12:14.867383 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:14.867347    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6lp\" (UniqueName: \"kubernetes.io/projected/b6934939-13dd-40c8-ae10-62971cf22838-kube-api-access-tb6lp\") pod \"payload-pre-processing-74584bb9f8-hsj5q\" (UID: \"b6934939-13dd-40c8-ae10-62971cf22838\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q"
Jul 14 01:12:15.049708 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:15.049649    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q"
Jul 14 01:12:15.191695 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:15.191665    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q"]
Jul 14 01:12:15.193802 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:12:15.193770    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6934939_13dd_40c8_ae10_62971cf22838.slice/crio-07044a6b582274b74d77a039e78b748b00c6102cdaf13dfbf7d20cafaddffb4a WatchSource:0}: Error finding container 07044a6b582274b74d77a039e78b748b00c6102cdaf13dfbf7d20cafaddffb4a: Status 404 returned error can't find the container with id 07044a6b582274b74d77a039e78b748b00c6102cdaf13dfbf7d20cafaddffb4a
Jul 14 01:12:16.083320 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:16.083278    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q" event={"ID":"b6934939-13dd-40c8-ae10-62971cf22838","Type":"ContainerStarted","Data":"07044a6b582274b74d77a039e78b748b00c6102cdaf13dfbf7d20cafaddffb4a"}
Jul 14 01:12:17.089975 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:17.089931    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q" event={"ID":"b6934939-13dd-40c8-ae10-62971cf22838","Type":"ContainerStarted","Data":"fc864c5bf956b58e734be6b78beb3c02a0f40b8e383d605f80c4a4dea3172b9f"}
Jul 14 01:12:17.090463 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:17.090063    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q"
Jul 14 01:12:17.105807 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:17.105749    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q" podStartSLOduration=1.7183970579999999 podStartE2EDuration="3.105731504s" totalImagesPullingTime="1.387334446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:12:17.103765002 +0000 UTC m=+766.668174056" watchObservedRunningTime="2026-07-14 01:12:17.105731504 +0000 UTC m=+766.670140558"
Jul 14 01:12:24.558831 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:24.558779    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:12:24.637717 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:24.637684    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:12:28.096900 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:28.096860    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q" podUID="b6934939-13dd-40c8-ae10-62971cf22838" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.36:9004: connect: connection refused"
Jul 14 01:12:37.859474 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:37.859430    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"]
Jul 14 01:12:37.864712 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:37.864681    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:37.866790 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:37.866743    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 14 01:12:37.867379 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:37.867353    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 14 01:12:37.867516 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:37.867422    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vmjgp\""
Jul 14 01:12:37.867582 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:37.867516    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 14 01:12:37.871631 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:37.871607    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"]
Jul 14 01:12:37.902482 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:37.902450    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:12:37.956017 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:37.955949    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dffd9eb6-560e-4ae1-9e09-84319ca10461-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:37.956205 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:37.956034    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dffd9eb6-560e-4ae1-9e09-84319ca10461-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:37.956205 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:37.956056    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dffd9eb6-560e-4ae1-9e09-84319ca10461-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:37.956205 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:37.956091    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dffd9eb6-560e-4ae1-9e09-84319ca10461-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:37.956205 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:37.956124    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dffd9eb6-560e-4ae1-9e09-84319ca10461-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:37.956205 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:37.956154    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2qn9\" (UniqueName: \"kubernetes.io/projected/dffd9eb6-560e-4ae1-9e09-84319ca10461-kube-api-access-r2qn9\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:38.056769 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:38.056728    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dffd9eb6-560e-4ae1-9e09-84319ca10461-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:38.056769 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:38.056773    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dffd9eb6-560e-4ae1-9e09-84319ca10461-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:38.057095 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:38.056799    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dffd9eb6-560e-4ae1-9e09-84319ca10461-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:38.057095 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:38.056849    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dffd9eb6-560e-4ae1-9e09-84319ca10461-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:38.057095 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:38.056878    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dffd9eb6-560e-4ae1-9e09-84319ca10461-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:38.057095 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:38.056922    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qn9\" (UniqueName: \"kubernetes.io/projected/dffd9eb6-560e-4ae1-9e09-84319ca10461-kube-api-access-r2qn9\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:38.057291 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:38.057236    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dffd9eb6-560e-4ae1-9e09-84319ca10461-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:38.057349 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:38.057304    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dffd9eb6-560e-4ae1-9e09-84319ca10461-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:38.057428 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:38.057406    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dffd9eb6-560e-4ae1-9e09-84319ca10461-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:38.059337 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:38.059318    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dffd9eb6-560e-4ae1-9e09-84319ca10461-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:38.059586 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:38.059569    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dffd9eb6-560e-4ae1-9e09-84319ca10461-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:38.064338 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:38.064310    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qn9\" (UniqueName: \"kubernetes.io/projected/dffd9eb6-560e-4ae1-9e09-84319ca10461-kube-api-access-r2qn9\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6\" (UID: \"dffd9eb6-560e-4ae1-9e09-84319ca10461\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:38.095633 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:38.095603    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-hsj5q"
Jul 14 01:12:38.176663 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:38.176619    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:38.322639 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:38.322610    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"]
Jul 14 01:12:38.324700 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:12:38.324673    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffd9eb6_560e_4ae1_9e09_84319ca10461.slice/crio-4b58862ece2859a4c220b446e01ae1ddc814bb200812baf6ad638dab797a5c5e WatchSource:0}: Error finding container 4b58862ece2859a4c220b446e01ae1ddc814bb200812baf6ad638dab797a5c5e: Status 404 returned error can't find the container with id 4b58862ece2859a4c220b446e01ae1ddc814bb200812baf6ad638dab797a5c5e
Jul 14 01:12:39.198468 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:39.198422    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6" event={"ID":"dffd9eb6-560e-4ae1-9e09-84319ca10461","Type":"ContainerStarted","Data":"4b58862ece2859a4c220b446e01ae1ddc814bb200812baf6ad638dab797a5c5e"}
Jul 14 01:12:40.205235 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:40.205197    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6" event={"ID":"dffd9eb6-560e-4ae1-9e09-84319ca10461","Type":"ContainerStarted","Data":"8e113f46c3533668c20c003d65e8f3b1df4efd5149640ef432d0d9f7e103c374"}
Jul 14 01:12:40.222719 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:40.222663    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6" podStartSLOduration=2.199220001 podStartE2EDuration="3.222646709s" totalImagesPullingTime="1.023426708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:12:40.220088721 +0000 UTC m=+789.784497777" watchObservedRunningTime="2026-07-14 01:12:40.222646709 +0000 UTC m=+789.787055800"
Jul 14 01:12:45.143683 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.143641    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"]
Jul 14 01:12:45.147526 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.147502    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.149559 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.149540    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 14 01:12:45.156862 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.156503    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"]
Jul 14 01:12:45.179013 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.178980    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:12:45.223165 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.223130    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47660f12-7c2a-455e-975d-213ebadf129b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.223332 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.223172    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47660f12-7c2a-455e-975d-213ebadf129b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.223332 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.223214    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6t24\" (UniqueName: \"kubernetes.io/projected/47660f12-7c2a-455e-975d-213ebadf129b-kube-api-access-x6t24\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.223332 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.223290    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47660f12-7c2a-455e-975d-213ebadf129b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.223458 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.223347    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47660f12-7c2a-455e-975d-213ebadf129b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.223458 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.223398    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47660f12-7c2a-455e-975d-213ebadf129b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.324610 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.324563    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47660f12-7c2a-455e-975d-213ebadf129b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.324610 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.324610    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47660f12-7c2a-455e-975d-213ebadf129b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.324865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.324644    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47660f12-7c2a-455e-975d-213ebadf129b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.324865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.324668    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6t24\" (UniqueName: \"kubernetes.io/projected/47660f12-7c2a-455e-975d-213ebadf129b-kube-api-access-x6t24\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.324865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.324718    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47660f12-7c2a-455e-975d-213ebadf129b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.324865 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.324790    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47660f12-7c2a-455e-975d-213ebadf129b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.325281 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.325255    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47660f12-7c2a-455e-975d-213ebadf129b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.327698 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.325442    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47660f12-7c2a-455e-975d-213ebadf129b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.327698 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.325480    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47660f12-7c2a-455e-975d-213ebadf129b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.331461 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.328298    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47660f12-7c2a-455e-975d-213ebadf129b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.331461 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.328659    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47660f12-7c2a-455e-975d-213ebadf129b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.333501 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.333474    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6t24\" (UniqueName: \"kubernetes.io/projected/47660f12-7c2a-455e-975d-213ebadf129b-kube-api-access-x6t24\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g6r55\" (UID: \"47660f12-7c2a-455e-975d-213ebadf129b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.461235 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.461136    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:45.600249 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:45.600221    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"]
Jul 14 01:12:45.602111 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:12:45.602083    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47660f12_7c2a_455e_975d_213ebadf129b.slice/crio-470a9afa612be7ca467906f8be1cd78f365bb2cf06f089f91536d545ee125041 WatchSource:0}: Error finding container 470a9afa612be7ca467906f8be1cd78f365bb2cf06f089f91536d545ee125041: Status 404 returned error can't find the container with id 470a9afa612be7ca467906f8be1cd78f365bb2cf06f089f91536d545ee125041
Jul 14 01:12:46.236364 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:46.236326    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55" event={"ID":"47660f12-7c2a-455e-975d-213ebadf129b","Type":"ContainerStarted","Data":"14eccece5c711c3ff07e869240659d107ce3a2d79a6277c6d295aa38851f8cc9"}
Jul 14 01:12:46.236768 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:46.236381    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55" event={"ID":"47660f12-7c2a-455e-975d-213ebadf129b","Type":"ContainerStarted","Data":"470a9afa612be7ca467906f8be1cd78f365bb2cf06f089f91536d545ee125041"}
Jul 14 01:12:46.256538 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:46.256470    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55" podStartSLOduration=1.071440324 podStartE2EDuration="1.256453448s" totalImagesPullingTime="185.013124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:12:46.253503621 +0000 UTC m=+795.817912676" watchObservedRunningTime="2026-07-14 01:12:46.256453448 +0000 UTC m=+795.820862503"
Jul 14 01:12:48.177782 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:48.177741    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:48.178213 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:48.177821    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:48.190558 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:48.190530    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:48.258051 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:48.258020    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6"
Jul 14 01:12:55.462131 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:55.462033    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:55.462131 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:55.462098    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:55.474601 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:55.474574    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:12:56.297248 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:12:56.297216    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g6r55"
Jul 14 01:13:15.886123 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:13:15.886088    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:13:23.875497 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:13:23.875462    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:13:30.706906 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:13:30.706872    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:13:36.778697 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:13:36.778658    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:13:49.881262 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:13:49.881224    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:13:49.931565 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:13:49.931529    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:14:05.667169 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:14:05.667123    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:14:05.767320 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:14:05.767278    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:14:45.804006 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:14:45.803944    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:14:45.956682 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:14:45.956643    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:14:46.998676 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:14:46.998640    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:14:47.093217 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:14:47.093185    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:15:00.143773 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:00.143734    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733195-kxvd4"]
Jul 14 01:15:00.147077 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:00.147060    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-kxvd4"
Jul 14 01:15:00.149115 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:00.149085    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kh8h4\""
Jul 14 01:15:00.153449 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:00.153424    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733195-kxvd4"]
Jul 14 01:15:00.157774 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:00.157750    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-kxvd4"
Jul 14 01:15:00.294919 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:00.294894    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733195-kxvd4"]
Jul 14 01:15:00.296496 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:15:00.296466    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93b7cd2_8875_45a2_9b55_5957fe44b7cb.slice/crio-1163f42b17cb3560e3b92450da44b5a9a634aac38343230441eeab92f3e222d1 WatchSource:0}: Error finding container 1163f42b17cb3560e3b92450da44b5a9a634aac38343230441eeab92f3e222d1: Status 404 returned error can't find the container with id 1163f42b17cb3560e3b92450da44b5a9a634aac38343230441eeab92f3e222d1
Jul 14 01:15:00.298469 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:00.298449    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 01:15:00.887852 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:00.887818    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-kxvd4" event={"ID":"d93b7cd2-8875-45a2-9b55-5957fe44b7cb","Type":"ContainerStarted","Data":"1163f42b17cb3560e3b92450da44b5a9a634aac38343230441eeab92f3e222d1"}
Jul 14 01:15:02.903187 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:02.903093    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-kxvd4" event={"ID":"d93b7cd2-8875-45a2-9b55-5957fe44b7cb","Type":"ContainerStarted","Data":"3cdaf8031d60305d9b67cce5ea5112e77b01a5d753e1010a7c0faf6bfcb054eb"}
Jul 14 01:15:02.917066 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:02.917015    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-kxvd4" podStartSLOduration=0.588045972 podStartE2EDuration="2.916998889s" totalImagesPullingTime="2.328952917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:15:02.914927269 +0000 UTC m=+932.479336406" watchObservedRunningTime="2026-07-14 01:15:02.916998889 +0000 UTC m=+932.481407967"
Jul 14 01:15:03.909742 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:03.909705    2567 generic.go:356] "Generic (PLEG): container finished" podID="d93b7cd2-8875-45a2-9b55-5957fe44b7cb" containerID="3cdaf8031d60305d9b67cce5ea5112e77b01a5d753e1010a7c0faf6bfcb054eb" exitCode=0
Jul 14 01:15:03.910250 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:03.909794    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-kxvd4" event={"ID":"d93b7cd2-8875-45a2-9b55-5957fe44b7cb","Type":"ContainerDied","Data":"3cdaf8031d60305d9b67cce5ea5112e77b01a5d753e1010a7c0faf6bfcb054eb"}
Jul 14 01:15:05.052968 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:05.052922    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-kxvd4"
Jul 14 01:15:05.923890 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:05.923858    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-kxvd4"
Jul 14 01:15:05.924129 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:05.923901    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-kxvd4" event={"ID":"d93b7cd2-8875-45a2-9b55-5957fe44b7cb","Type":"ContainerDied","Data":"1163f42b17cb3560e3b92450da44b5a9a634aac38343230441eeab92f3e222d1"}
Jul 14 01:15:05.924129 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:05.923947    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1163f42b17cb3560e3b92450da44b5a9a634aac38343230441eeab92f3e222d1"
Jul 14 01:15:18.563499 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:18.563461    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:15:20.467847 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:20.467810    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:15:38.866909 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:38.866873    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:15:45.556890 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:15:45.556857    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:16:02.183816 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:16:02.183721    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:16:03.595708 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:16:03.595667    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:16:39.942717 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:16:39.942676    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:16:39.993921 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:16:39.993881    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:16:43.158575 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:16:43.158536    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:16:43.257842 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:16:43.257795    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:16:43.425030 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:16:43.424922    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:16:49.064558 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:16:49.064526    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:16:49.164769 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:16:49.164722    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:16:49.265822 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:16:49.265794    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:16:49.432802 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:16:49.432768    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:16:55.236886 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:16:55.236847    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:16:55.293541 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:16:55.293507    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:18:45.276950 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:18:45.276908    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:20:22.389580 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:20:22.389499    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:23:00.882580 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:23:00.882539    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:23:00.935298 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:23:00.935259    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:24:06.870135 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:24:06.870100    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:27:57.831790 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:27:57.831705    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:05.267191 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:05.267147    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:13.532280 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:13.532240    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:18.668554 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:18.668519    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:21.670945 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:21.670913    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:21.774988 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:21.774924    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:22.145573 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:22.145540    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:27.782110 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:27.782077    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:27.846180 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:27.846149    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:27.950990 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:27.950858    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"]
Jul 14 01:28:27.954735 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:27.951343    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d93b7cd2-8875-45a2-9b55-5957fe44b7cb" containerName="cleanup"
Jul 14 01:28:27.954735 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:27.951366    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d93b7cd2-8875-45a2-9b55-5957fe44b7cb" containerName="cleanup"
Jul 14 01:28:27.954735 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:27.951481    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d93b7cd2-8875-45a2-9b55-5957fe44b7cb" containerName="cleanup"
Jul 14 01:28:27.956625 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:27.956600    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:27.958825 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:27.958800    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-62a42e82-openshift-default-dockercfg-jj6v4\""
Jul 14 01:28:27.973696 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:27.973669    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:28.009745 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.009706    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"]
Jul 14 01:28:28.062066 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.061980    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-token\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.062066 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.062026    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-workload-certs\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.062066 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.062048    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-data\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.062350 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.062151    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-workload-socket\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.062350 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.062193    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-podinfo\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.062350 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.062227    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istiod-ca-cert\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.062350 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.062262    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-envoy\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.062350 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.062302    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrd6r\" (UniqueName: \"kubernetes.io/projected/01c04727-0997-4fbc-8d74-c13dfb40c3b6-kube-api-access-wrd6r\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.062350 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.062329    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-credential-socket\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.073268 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.073236    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:28.163036 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.162982    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-token\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.163036 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.163041    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-workload-certs\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.163284 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.163062    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-data\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.163284 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.163086    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-workload-socket\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.163284 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.163102    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-podinfo\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.163284 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.163126    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istiod-ca-cert\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.163284 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.163147    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-envoy\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.163284 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.163168    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrd6r\" (UniqueName: \"kubernetes.io/projected/01c04727-0997-4fbc-8d74-c13dfb40c3b6-kube-api-access-wrd6r\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.163284 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.163202    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-credential-socket\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.163622 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.163523    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-data\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.163659 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.163616    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-workload-socket\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.163693 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.163669    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-workload-certs\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.163728 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.163686    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-credential-socket\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.163907 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.163889    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istiod-ca-cert\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.165549 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.165529    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-envoy\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.166068 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.166051    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-podinfo\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.170730 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.170709    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-token\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.172118 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.172088    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrd6r\" (UniqueName: \"kubernetes.io/projected/01c04727-0997-4fbc-8d74-c13dfb40c3b6-kube-api-access-wrd6r\") pod \"e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") " pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.268563 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.268512    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:28.412395 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.412360    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"]
Jul 14 01:28:28.414397 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:28:28.414373    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c04727_0997_4fbc_8d74_c13dfb40c3b6.slice/crio-f254def84a5a78715def39d63f24a9b9bee7b0f6730696ebcb4fa4169745064a WatchSource:0}: Error finding container f254def84a5a78715def39d63f24a9b9bee7b0f6730696ebcb4fa4169745064a: Status 404 returned error can't find the container with id f254def84a5a78715def39d63f24a9b9bee7b0f6730696ebcb4fa4169745064a
Jul 14 01:28:28.416755 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.416715    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:28:28.416844 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.416810    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:28:28.416884 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.416850    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:28:28.932637 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.932598    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k" event={"ID":"01c04727-0997-4fbc-8d74-c13dfb40c3b6","Type":"ContainerStarted","Data":"aec28ed5eeb1470fb1a95f0ebdea1f070e17ca932a1015e593f2ba58243c9fd5"}
Jul 14 01:28:28.933190 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.932651    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k" event={"ID":"01c04727-0997-4fbc-8d74-c13dfb40c3b6","Type":"ContainerStarted","Data":"f254def84a5a78715def39d63f24a9b9bee7b0f6730696ebcb4fa4169745064a"}
Jul 14 01:28:28.950438 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:28.950376    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k" podStartSLOduration=1.950354909 podStartE2EDuration="1.950354909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:28:28.949376124 +0000 UTC m=+1738.513785179" watchObservedRunningTime="2026-07-14 01:28:28.950354909 +0000 UTC m=+1738.514763966"
Jul 14 01:28:29.078637 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:29.078597    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:29.170826 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:29.170793    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:29.235203 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:29.235111    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:29.269202 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:29.269167    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:29.297643 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:29.297610    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:29.344378 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:29.344345    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:29.726224 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:29.726185    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:29.827436 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:29.827400    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:29.938247 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:29.938214    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:29.939407 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:29.939387    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:31.763179 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:31.763148    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:28:34.176978 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:34.176924    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf"]
Jul 14 01:28:34.180528 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:34.180509    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf"
Jul 14 01:28:34.182588 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:34.182563    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kh8h4\""
Jul 14 01:28:34.182714 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:34.182563    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:28:34.186048 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:34.186024    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf"]
Jul 14 01:28:34.217768 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:34.217728    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42ba017-94eb-42c7-88aa-9eff62dd658c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf\" (UID: \"f42ba017-94eb-42c7-88aa-9eff62dd658c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf"
Jul 14 01:28:34.319200 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:34.319152    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42ba017-94eb-42c7-88aa-9eff62dd658c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf\" (UID: \"f42ba017-94eb-42c7-88aa-9eff62dd658c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf"
Jul 14 01:28:34.319761 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:34.319741    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42ba017-94eb-42c7-88aa-9eff62dd658c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf\" (UID: \"f42ba017-94eb-42c7-88aa-9eff62dd658c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf"
Jul 14 01:28:34.492926 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:34.492832    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf"
Jul 14 01:28:34.662662 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:34.657853    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf"]
Jul 14 01:28:34.663772 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:28:34.663727    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42ba017_94eb_42c7_88aa_9eff62dd658c.slice/crio-10f7d18fc00d2afd5067e7af3087badb6433d1c1873d0e92137f1ed14b30d40c WatchSource:0}: Error finding container 10f7d18fc00d2afd5067e7af3087badb6433d1c1873d0e92137f1ed14b30d40c: Status 404 returned error can't find the container with id 10f7d18fc00d2afd5067e7af3087badb6433d1c1873d0e92137f1ed14b30d40c
Jul 14 01:28:34.977386 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:34.977351    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf" event={"ID":"f42ba017-94eb-42c7-88aa-9eff62dd658c","Type":"ContainerStarted","Data":"caa88186bcf3afb329c0c2f49807b96547901a2d6eca4f1e738bd7535765fd21"}
Jul 14 01:28:34.977556 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:34.977412    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf" event={"ID":"f42ba017-94eb-42c7-88aa-9eff62dd658c","Type":"ContainerStarted","Data":"10f7d18fc00d2afd5067e7af3087badb6433d1c1873d0e92137f1ed14b30d40c"}
Jul 14 01:28:34.982462 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:34.982383    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf" podStartSLOduration=0.982362761 podStartE2EDuration="982.362761ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:28:34.981891749 +0000 UTC m=+1744.546300806" watchObservedRunningTime="2026-07-14 01:28:34.982362761 +0000 UTC m=+1744.546771817"
Jul 14 01:28:35.977890 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:35.977858    2567 generic.go:356] "Generic (PLEG): container finished" podID="f42ba017-94eb-42c7-88aa-9eff62dd658c" containerID="caa88186bcf3afb329c0c2f49807b96547901a2d6eca4f1e738bd7535765fd21" exitCode=7
Jul 14 01:28:35.978315 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:35.977913    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf" event={"ID":"f42ba017-94eb-42c7-88aa-9eff62dd658c","Type":"ContainerDied","Data":"caa88186bcf3afb329c0c2f49807b96547901a2d6eca4f1e738bd7535765fd21"}
Jul 14 01:28:35.978315 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:35.978165    2567 scope.go:117] "RemoveContainer" containerID="caa88186bcf3afb329c0c2f49807b96547901a2d6eca4f1e738bd7535765fd21"
Jul 14 01:28:36.985082 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:36.985046    2567 generic.go:356] "Generic (PLEG): container finished" podID="f42ba017-94eb-42c7-88aa-9eff62dd658c" containerID="fc205e85a377c3fac1c58b98b5c948d2b0498f4f57738cb9350c9d332d3cbf80" exitCode=7
Jul 14 01:28:36.985518 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:36.985142    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf" event={"ID":"f42ba017-94eb-42c7-88aa-9eff62dd658c","Type":"ContainerDied","Data":"fc205e85a377c3fac1c58b98b5c948d2b0498f4f57738cb9350c9d332d3cbf80"}
Jul 14 01:28:36.985518 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:36.985205    2567 scope.go:117] "RemoveContainer" containerID="caa88186bcf3afb329c0c2f49807b96547901a2d6eca4f1e738bd7535765fd21"
Jul 14 01:28:36.985518 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:36.985393    2567 scope.go:117] "RemoveContainer" containerID="fc205e85a377c3fac1c58b98b5c948d2b0498f4f57738cb9350c9d332d3cbf80"
Jul 14 01:28:36.985659 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:28:36.985596    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf_odh-ai-gateway-infra(f42ba017-94eb-42c7-88aa-9eff62dd658c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf" podUID="f42ba017-94eb-42c7-88aa-9eff62dd658c"
Jul 14 01:28:37.993333 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:37.993305    2567 scope.go:117] "RemoveContainer" containerID="fc205e85a377c3fac1c58b98b5c948d2b0498f4f57738cb9350c9d332d3cbf80"
Jul 14 01:28:37.993712 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:28:37.993524    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf_odh-ai-gateway-infra(f42ba017-94eb-42c7-88aa-9eff62dd658c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf" podUID="f42ba017-94eb-42c7-88aa-9eff62dd658c"
Jul 14 01:28:41.011147 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:41.011112    2567 generic.go:356] "Generic (PLEG): container finished" podID="01c04727-0997-4fbc-8d74-c13dfb40c3b6" containerID="aec28ed5eeb1470fb1a95f0ebdea1f070e17ca932a1015e593f2ba58243c9fd5" exitCode=0
Jul 14 01:28:41.011569 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:41.011170    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k" event={"ID":"01c04727-0997-4fbc-8d74-c13dfb40c3b6","Type":"ContainerDied","Data":"aec28ed5eeb1470fb1a95f0ebdea1f070e17ca932a1015e593f2ba58243c9fd5"}
Jul 14 01:28:41.011569 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:41.011563    2567 scope.go:117] "RemoveContainer" containerID="aec28ed5eeb1470fb1a95f0ebdea1f070e17ca932a1015e593f2ba58243c9fd5"
Jul 14 01:28:41.012641 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:41.012611    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:28:41.012721 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:41.012699    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:28:41.012761 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:41.012742    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:28:42.024529 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:42.024495    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k" event={"ID":"01c04727-0997-4fbc-8d74-c13dfb40c3b6","Type":"ContainerStarted","Data":"05caf04715ca0943ef3e3ff318a8d0a9a06e13322dbdd74ecf762003adca8145"}
Jul 14 01:28:42.268921 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:42.268877    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:42.270215 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:42.270187    2567 patch_prober.go:28] interesting pod/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 14 01:28:42.270353 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:42.270237    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k" podUID="01c04727-0997-4fbc-8d74-c13dfb40c3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 14 01:28:43.269787 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:43.269749    2567 patch_prober.go:28] interesting pod/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 14 01:28:43.270222 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:43.269814    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k" podUID="01c04727-0997-4fbc-8d74-c13dfb40c3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 14 01:28:44.268981 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:44.268917    2567 patch_prober.go:28] interesting pod/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 14 01:28:44.269214 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:44.269005    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k" podUID="01c04727-0997-4fbc-8d74-c13dfb40c3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 14 01:28:44.939054 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:44.939006    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:45.269075 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:45.268980    2567 patch_prober.go:28] interesting pod/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 14 01:28:45.269075 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:45.269059    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k" podUID="01c04727-0997-4fbc-8d74-c13dfb40c3b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 14 01:28:46.273187 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:46.273147    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:46.274321 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:46.274302    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:28:52.969838 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:52.969809    2567 scope.go:117] "RemoveContainer" containerID="fc205e85a377c3fac1c58b98b5c948d2b0498f4f57738cb9350c9d332d3cbf80"
Jul 14 01:28:54.092007 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:54.091892    2567 generic.go:356] "Generic (PLEG): container finished" podID="f42ba017-94eb-42c7-88aa-9eff62dd658c" containerID="e8947eeee93a99fce65035c283c39ebf441b84f1646caa954a7944be2ead1797" exitCode=0
Jul 14 01:28:54.092007 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:54.091942    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf" event={"ID":"f42ba017-94eb-42c7-88aa-9eff62dd658c","Type":"ContainerDied","Data":"e8947eeee93a99fce65035c283c39ebf441b84f1646caa954a7944be2ead1797"}
Jul 14 01:28:54.092428 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:54.092009    2567 scope.go:117] "RemoveContainer" containerID="fc205e85a377c3fac1c58b98b5c948d2b0498f4f57738cb9350c9d332d3cbf80"
Jul 14 01:28:55.122676 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:55.122643    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf"]
Jul 14 01:28:55.240151 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:55.240124    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf"
Jul 14 01:28:55.317086 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:55.317050    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42ba017-94eb-42c7-88aa-9eff62dd658c-maas-api-service-ca\") pod \"f42ba017-94eb-42c7-88aa-9eff62dd658c\" (UID: \"f42ba017-94eb-42c7-88aa-9eff62dd658c\") "
Jul 14 01:28:55.317418 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:55.317396    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42ba017-94eb-42c7-88aa-9eff62dd658c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f42ba017-94eb-42c7-88aa-9eff62dd658c" (UID: "f42ba017-94eb-42c7-88aa-9eff62dd658c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:28:55.418011 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:55.417975    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42ba017-94eb-42c7-88aa-9eff62dd658c-maas-api-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:28:56.104641 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:56.104607    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf" event={"ID":"f42ba017-94eb-42c7-88aa-9eff62dd658c","Type":"ContainerDied","Data":"10f7d18fc00d2afd5067e7af3087badb6433d1c1873d0e92137f1ed14b30d40c"}
Jul 14 01:28:56.104825 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:56.104658    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10f7d18fc00d2afd5067e7af3087badb6433d1c1873d0e92137f1ed14b30d40c"
Jul 14 01:28:56.104825 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:56.104614    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf"
Jul 14 01:28:56.132567 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:56.132529    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf"]
Jul 14 01:28:56.135913 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:56.135880    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-ks4gf"]
Jul 14 01:28:56.974344 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:28:56.974308    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f42ba017-94eb-42c7-88aa-9eff62dd658c" path="/var/lib/kubelet/pods/f42ba017-94eb-42c7-88aa-9eff62dd658c/volumes"
Jul 14 01:29:34.251200 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.251162    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn"]
Jul 14 01:29:34.251635 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.251503    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f42ba017-94eb-42c7-88aa-9eff62dd658c" containerName="revoke-keys"
Jul 14 01:29:34.251635 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.251515    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f42ba017-94eb-42c7-88aa-9eff62dd658c" containerName="revoke-keys"
Jul 14 01:29:34.251635 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.251528    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f42ba017-94eb-42c7-88aa-9eff62dd658c" containerName="revoke-keys"
Jul 14 01:29:34.251635 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.251534    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f42ba017-94eb-42c7-88aa-9eff62dd658c" containerName="revoke-keys"
Jul 14 01:29:34.251635 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.251594    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f42ba017-94eb-42c7-88aa-9eff62dd658c" containerName="revoke-keys"
Jul 14 01:29:34.251635 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.251602    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f42ba017-94eb-42c7-88aa-9eff62dd658c" containerName="revoke-keys"
Jul 14 01:29:34.251635 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.251610    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f42ba017-94eb-42c7-88aa-9eff62dd658c" containerName="revoke-keys"
Jul 14 01:29:34.254543 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.254525    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn"
Jul 14 01:29:34.256450 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.256430    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:29:34.256567 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.256507    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kh8h4\""
Jul 14 01:29:34.261885 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.261850    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn"]
Jul 14 01:29:34.365135 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.365092    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14c59f6b-e7e1-4144-b170-1389de6a0c01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn\" (UID: \"14c59f6b-e7e1-4144-b170-1389de6a0c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn"
Jul 14 01:29:34.466010 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.465941    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14c59f6b-e7e1-4144-b170-1389de6a0c01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn\" (UID: \"14c59f6b-e7e1-4144-b170-1389de6a0c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn"
Jul 14 01:29:34.466631 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.466603    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14c59f6b-e7e1-4144-b170-1389de6a0c01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn\" (UID: \"14c59f6b-e7e1-4144-b170-1389de6a0c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn"
Jul 14 01:29:34.566869 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.566773    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn"
Jul 14 01:29:34.704795 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:34.704764    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn"]
Jul 14 01:29:34.706564 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:29:34.706527    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c59f6b_e7e1_4144_b170_1389de6a0c01.slice/crio-0e82cb62a660bea1e4757fd85c444648d3e327c7465463d785bb804f08dc9b76 WatchSource:0}: Error finding container 0e82cb62a660bea1e4757fd85c444648d3e327c7465463d785bb804f08dc9b76: Status 404 returned error can't find the container with id 0e82cb62a660bea1e4757fd85c444648d3e327c7465463d785bb804f08dc9b76
Jul 14 01:29:35.304556 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:35.304430    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn" event={"ID":"14c59f6b-e7e1-4144-b170-1389de6a0c01","Type":"ContainerStarted","Data":"45b66b8c81db3e1a2a7adad51eea76bed4b6c372e00e490551847e8b2dec1acb"}
Jul 14 01:29:35.304556 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:35.304526    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn" event={"ID":"14c59f6b-e7e1-4144-b170-1389de6a0c01","Type":"ContainerStarted","Data":"0e82cb62a660bea1e4757fd85c444648d3e327c7465463d785bb804f08dc9b76"}
Jul 14 01:29:35.320171 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:35.320110    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn" podStartSLOduration=1.320094912 podStartE2EDuration="1.320094912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:29:35.316930696 +0000 UTC m=+1804.881339749" watchObservedRunningTime="2026-07-14 01:29:35.320094912 +0000 UTC m=+1804.884503966"
Jul 14 01:29:36.310737 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:36.310704    2567 generic.go:356] "Generic (PLEG): container finished" podID="14c59f6b-e7e1-4144-b170-1389de6a0c01" containerID="45b66b8c81db3e1a2a7adad51eea76bed4b6c372e00e490551847e8b2dec1acb" exitCode=0
Jul 14 01:29:36.311144 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:36.310764    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn" event={"ID":"14c59f6b-e7e1-4144-b170-1389de6a0c01","Type":"ContainerDied","Data":"45b66b8c81db3e1a2a7adad51eea76bed4b6c372e00e490551847e8b2dec1acb"}
Jul 14 01:29:37.452224 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:37.452197    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn"
Jul 14 01:29:37.592474 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:37.592377    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14c59f6b-e7e1-4144-b170-1389de6a0c01-maas-api-service-ca\") pod \"14c59f6b-e7e1-4144-b170-1389de6a0c01\" (UID: \"14c59f6b-e7e1-4144-b170-1389de6a0c01\") "
Jul 14 01:29:37.592657 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:37.592636    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c59f6b-e7e1-4144-b170-1389de6a0c01-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14c59f6b-e7e1-4144-b170-1389de6a0c01" (UID: "14c59f6b-e7e1-4144-b170-1389de6a0c01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:29:37.693510 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:37.693462    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14c59f6b-e7e1-4144-b170-1389de6a0c01-maas-api-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:29:38.323490 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:38.323399    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn"
Jul 14 01:29:38.323490 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:38.323437    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn" event={"ID":"14c59f6b-e7e1-4144-b170-1389de6a0c01","Type":"ContainerDied","Data":"0e82cb62a660bea1e4757fd85c444648d3e327c7465463d785bb804f08dc9b76"}
Jul 14 01:29:38.323490 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:38.323485    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e82cb62a660bea1e4757fd85c444648d3e327c7465463d785bb804f08dc9b76"
Jul 14 01:29:44.638134 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:44.638098    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:29:49.302301 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:49.302266    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:29:54.922646 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:54.922598    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:29:55.024761 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:55.024697    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:29:59.487832 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:59.487772    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"]
Jul 14 01:29:59.488339 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:59.488105    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k" podUID="01c04727-0997-4fbc-8d74-c13dfb40c3b6" containerName="istio-proxy" containerID="cri-o://05caf04715ca0943ef3e3ff318a8d0a9a06e13322dbdd74ecf762003adca8145" gracePeriod=30
Jul 14 01:29:59.515175 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:59.515137    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:29:59.888749 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:59.888712    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:29:59.988748 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:29:59.988709    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:30:00.060091 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.060052    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"]
Jul 14 01:30:00.060573 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.060558    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f42ba017-94eb-42c7-88aa-9eff62dd658c" containerName="revoke-keys"
Jul 14 01:30:00.060636 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.060577    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f42ba017-94eb-42c7-88aa-9eff62dd658c" containerName="revoke-keys"
Jul 14 01:30:00.060636 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.060599    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14c59f6b-e7e1-4144-b170-1389de6a0c01" containerName="revoke-keys"
Jul 14 01:30:00.060636 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.060608    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="14c59f6b-e7e1-4144-b170-1389de6a0c01" containerName="revoke-keys"
Jul 14 01:30:00.060734 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.060691    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="14c59f6b-e7e1-4144-b170-1389de6a0c01" containerName="revoke-keys"
Jul 14 01:30:00.064233 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.064211    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.066266 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.066240    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7149234b-gw-openshift-default-dockercfg-5ct8f\""
Jul 14 01:30:00.074991 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.074940    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"]
Jul 14 01:30:00.088062 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.087997    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-workload-certs\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.088062 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.088038    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-credential-socket\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.088578 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.088063    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-token\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.088578 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.088167    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istiod-ca-cert\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.088578 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.088206    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-data\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.088578 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.088239    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwzdr\" (UniqueName: \"kubernetes.io/projected/03882f99-5bbb-4ec8-9cfc-febddbb71b70-kube-api-access-pwzdr\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.088578 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.088277    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-podinfo\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.088578 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.088311    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-workload-socket\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.088578 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.088338    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-envoy\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.105394 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.105358    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:30:00.188904 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.188816    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-podinfo\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.188904 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.188857    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-workload-socket\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.188904 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.188875    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-envoy\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.189296 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.188921    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-workload-certs\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.189296 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.188976    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-credential-socket\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.189296 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.189003    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-token\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.189296 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.189057    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istiod-ca-cert\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.189296 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.189095    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-data\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.189296 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.189134    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwzdr\" (UniqueName: \"kubernetes.io/projected/03882f99-5bbb-4ec8-9cfc-febddbb71b70-kube-api-access-pwzdr\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.189648 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.189331    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-workload-socket\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.189648 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.189463    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-credential-socket\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.189648 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.189535    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-workload-certs\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.189648 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.189588    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-data\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.189884 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.189868    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istiod-ca-cert\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.191490 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.191470    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-envoy\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.191721 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.191704    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-podinfo\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.201554 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.201521    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-token\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.202343 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.202320    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwzdr\" (UniqueName: \"kubernetes.io/projected/03882f99-5bbb-4ec8-9cfc-febddbb71b70-kube-api-access-pwzdr\") pod \"e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") " pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.377399 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.377352    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:00.521502 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.521145    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"]
Jul 14 01:30:00.524245 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:30:00.524212    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03882f99_5bbb_4ec8_9cfc_febddbb71b70.slice/crio-81e8dfea9fe6a1e248b8774a9a9d8860dce43c1bcfcbf54ebd7291699202e599 WatchSource:0}: Error finding container 81e8dfea9fe6a1e248b8774a9a9d8860dce43c1bcfcbf54ebd7291699202e599: Status 404 returned error can't find the container with id 81e8dfea9fe6a1e248b8774a9a9d8860dce43c1bcfcbf54ebd7291699202e599
Jul 14 01:30:00.526380 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.526348    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:30:00.526479 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.526415    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:30:00.526479 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:00.526447    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:30:01.276260 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:01.276224    2567 patch_prober.go:28] interesting pod/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 01:30:01.276434 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:01.276285    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k" podUID="01c04727-0997-4fbc-8d74-c13dfb40c3b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 01:30:01.414899 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:01.414866    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:30:01.451340 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:01.451307    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm" event={"ID":"03882f99-5bbb-4ec8-9cfc-febddbb71b70","Type":"ContainerStarted","Data":"c60a025334d678fad09a800ccc345fa79355ac8d6a15ec79f67d3b50354604d0"}
Jul 14 01:30:01.451521 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:01.451363    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm" event={"ID":"03882f99-5bbb-4ec8-9cfc-febddbb71b70","Type":"ContainerStarted","Data":"81e8dfea9fe6a1e248b8774a9a9d8860dce43c1bcfcbf54ebd7291699202e599"}
Jul 14 01:30:01.470107 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:01.470054    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm" podStartSLOduration=1.470035953 podStartE2EDuration="1.470035953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:30:01.468844037 +0000 UTC m=+1831.033253095" watchObservedRunningTime="2026-07-14 01:30:01.470035953 +0000 UTC m=+1831.034445008"
Jul 14 01:30:01.574790 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:01.574692    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:30:01.802342 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:01.802309    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:30:02.039539 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:02.039507    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:30:02.140106 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:02.140068    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:30:02.377677 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:02.377644    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:02.382749 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:02.382725    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:02.457112 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:02.457080    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:02.458149 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:02.458121    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:04.741845 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.741819    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:30:04.825673 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.825648    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-token\") pod \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") "
Jul 14 01:30:04.825854 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.825690    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-credential-socket\") pod \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") "
Jul 14 01:30:04.825854 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.825717    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-envoy\") pod \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") "
Jul 14 01:30:04.825854 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.825737    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-workload-certs\") pod \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") "
Jul 14 01:30:04.825854 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.825760    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istiod-ca-cert\") pod \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") "
Jul 14 01:30:04.825854 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.825791    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-podinfo\") pod \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") "
Jul 14 01:30:04.825854 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.825812    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-workload-socket\") pod \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") "
Jul 14 01:30:04.825854 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.825842    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-data\") pod \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") "
Jul 14 01:30:04.826241 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.825862    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrd6r\" (UniqueName: \"kubernetes.io/projected/01c04727-0997-4fbc-8d74-c13dfb40c3b6-kube-api-access-wrd6r\") pod \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\" (UID: \"01c04727-0997-4fbc-8d74-c13dfb40c3b6\") "
Jul 14 01:30:04.826241 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.826008    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "01c04727-0997-4fbc-8d74-c13dfb40c3b6" (UID: "01c04727-0997-4fbc-8d74-c13dfb40c3b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:30:04.826241 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.826236    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-credential-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:30:04.826398 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.826322    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "01c04727-0997-4fbc-8d74-c13dfb40c3b6" (UID: "01c04727-0997-4fbc-8d74-c13dfb40c3b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:30:04.826586 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.826561    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "01c04727-0997-4fbc-8d74-c13dfb40c3b6" (UID: "01c04727-0997-4fbc-8d74-c13dfb40c3b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:30:04.826687 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.826656    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "01c04727-0997-4fbc-8d74-c13dfb40c3b6" (UID: "01c04727-0997-4fbc-8d74-c13dfb40c3b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:30:04.827032 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.827004    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "01c04727-0997-4fbc-8d74-c13dfb40c3b6" (UID: "01c04727-0997-4fbc-8d74-c13dfb40c3b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:30:04.828432 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.828410    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c04727-0997-4fbc-8d74-c13dfb40c3b6-kube-api-access-wrd6r" (OuterVolumeSpecName: "kube-api-access-wrd6r") pod "01c04727-0997-4fbc-8d74-c13dfb40c3b6" (UID: "01c04727-0997-4fbc-8d74-c13dfb40c3b6"). InnerVolumeSpecName "kube-api-access-wrd6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:30:04.828565 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.828519    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "01c04727-0997-4fbc-8d74-c13dfb40c3b6" (UID: "01c04727-0997-4fbc-8d74-c13dfb40c3b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:30:04.828565 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.828543    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "01c04727-0997-4fbc-8d74-c13dfb40c3b6" (UID: "01c04727-0997-4fbc-8d74-c13dfb40c3b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 01:30:04.828904 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.828883    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "01c04727-0997-4fbc-8d74-c13dfb40c3b6" (UID: "01c04727-0997-4fbc-8d74-c13dfb40c3b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:30:04.926869 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.926827    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-podinfo\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:30:04.926869 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.926868    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-workload-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:30:04.927138 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.926882    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-data\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:30:04.927138 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.926894    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrd6r\" (UniqueName: \"kubernetes.io/projected/01c04727-0997-4fbc-8d74-c13dfb40c3b6-kube-api-access-wrd6r\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:30:04.927138 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.926907    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-token\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:30:04.927138 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.926920    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istio-envoy\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:30:04.927138 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.926930    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01c04727-0997-4fbc-8d74-c13dfb40c3b6-workload-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:30:04.927138 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:04.926939    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01c04727-0997-4fbc-8d74-c13dfb40c3b6-istiod-ca-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:30:05.474519 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.474482    2567 generic.go:356] "Generic (PLEG): container finished" podID="01c04727-0997-4fbc-8d74-c13dfb40c3b6" containerID="05caf04715ca0943ef3e3ff318a8d0a9a06e13322dbdd74ecf762003adca8145" exitCode=0
Jul 14 01:30:05.474691 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.474587    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"
Jul 14 01:30:05.474691 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.474601    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k" event={"ID":"01c04727-0997-4fbc-8d74-c13dfb40c3b6","Type":"ContainerDied","Data":"05caf04715ca0943ef3e3ff318a8d0a9a06e13322dbdd74ecf762003adca8145"}
Jul 14 01:30:05.474691 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.474645    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k" event={"ID":"01c04727-0997-4fbc-8d74-c13dfb40c3b6","Type":"ContainerDied","Data":"f254def84a5a78715def39d63f24a9b9bee7b0f6730696ebcb4fa4169745064a"}
Jul 14 01:30:05.474691 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.474667    2567 scope.go:117] "RemoveContainer" containerID="05caf04715ca0943ef3e3ff318a8d0a9a06e13322dbdd74ecf762003adca8145"
Jul 14 01:30:05.486177 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.486155    2567 scope.go:117] "RemoveContainer" containerID="aec28ed5eeb1470fb1a95f0ebdea1f070e17ca932a1015e593f2ba58243c9fd5"
Jul 14 01:30:05.490994 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.490969    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"]
Jul 14 01:30:05.495511 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.495489    2567 scope.go:117] "RemoveContainer" containerID="05caf04715ca0943ef3e3ff318a8d0a9a06e13322dbdd74ecf762003adca8145"
Jul 14 01:30:05.495803 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:30:05.495782    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05caf04715ca0943ef3e3ff318a8d0a9a06e13322dbdd74ecf762003adca8145\": container with ID starting with 05caf04715ca0943ef3e3ff318a8d0a9a06e13322dbdd74ecf762003adca8145 not found: ID does not exist" containerID="05caf04715ca0943ef3e3ff318a8d0a9a06e13322dbdd74ecf762003adca8145"
Jul 14 01:30:05.495867 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.495813    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05caf04715ca0943ef3e3ff318a8d0a9a06e13322dbdd74ecf762003adca8145"} err="failed to get container status \"05caf04715ca0943ef3e3ff318a8d0a9a06e13322dbdd74ecf762003adca8145\": rpc error: code = NotFound desc = could not find container \"05caf04715ca0943ef3e3ff318a8d0a9a06e13322dbdd74ecf762003adca8145\": container with ID starting with 05caf04715ca0943ef3e3ff318a8d0a9a06e13322dbdd74ecf762003adca8145 not found: ID does not exist"
Jul 14 01:30:05.495867 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.495840    2567 scope.go:117] "RemoveContainer" containerID="aec28ed5eeb1470fb1a95f0ebdea1f070e17ca932a1015e593f2ba58243c9fd5"
Jul 14 01:30:05.496074 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:30:05.496051    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec28ed5eeb1470fb1a95f0ebdea1f070e17ca932a1015e593f2ba58243c9fd5\": container with ID starting with aec28ed5eeb1470fb1a95f0ebdea1f070e17ca932a1015e593f2ba58243c9fd5 not found: ID does not exist" containerID="aec28ed5eeb1470fb1a95f0ebdea1f070e17ca932a1015e593f2ba58243c9fd5"
Jul 14 01:30:05.496136 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.496076    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec28ed5eeb1470fb1a95f0ebdea1f070e17ca932a1015e593f2ba58243c9fd5"} err="failed to get container status \"aec28ed5eeb1470fb1a95f0ebdea1f070e17ca932a1015e593f2ba58243c9fd5\": rpc error: code = NotFound desc = could not find container \"aec28ed5eeb1470fb1a95f0ebdea1f070e17ca932a1015e593f2ba58243c9fd5\": container with ID starting with aec28ed5eeb1470fb1a95f0ebdea1f070e17ca932a1015e593f2ba58243c9fd5 not found: ID does not exist"
Jul 14 01:30:05.496722 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.496703    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-62a42e82-openshift-default-77f8bdd987-b5j9k"]
Jul 14 01:30:05.847377 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.847276    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp"]
Jul 14 01:30:05.847732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.847612    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c04727-0997-4fbc-8d74-c13dfb40c3b6" containerName="istio-proxy"
Jul 14 01:30:05.847732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.847624    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c04727-0997-4fbc-8d74-c13dfb40c3b6" containerName="istio-proxy"
Jul 14 01:30:05.847732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.847643    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c04727-0997-4fbc-8d74-c13dfb40c3b6" containerName="istio-proxy"
Jul 14 01:30:05.847732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.847649    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c04727-0997-4fbc-8d74-c13dfb40c3b6" containerName="istio-proxy"
Jul 14 01:30:05.847732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.847703    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="01c04727-0997-4fbc-8d74-c13dfb40c3b6" containerName="istio-proxy"
Jul 14 01:30:05.852352 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.852329    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp"
Jul 14 01:30:05.854391 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.854365    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:30:05.854543 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.854387    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kh8h4\""
Jul 14 01:30:05.856900 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.856876    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp"]
Jul 14 01:30:05.933102 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:05.933067    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/332b0bac-8489-4d8e-9680-853333551579-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7149234b-vskwp\" (UID: \"332b0bac-8489-4d8e-9680-853333551579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp"
Jul 14 01:30:06.033633 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:06.033601    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/332b0bac-8489-4d8e-9680-853333551579-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7149234b-vskwp\" (UID: \"332b0bac-8489-4d8e-9680-853333551579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp"
Jul 14 01:30:06.034631 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:06.034610    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/332b0bac-8489-4d8e-9680-853333551579-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7149234b-vskwp\" (UID: \"332b0bac-8489-4d8e-9680-853333551579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp"
Jul 14 01:30:06.103533 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:06.103492    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:30:06.165499 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:06.165448    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp"
Jul 14 01:30:06.205196 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:06.205158    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:30:06.309706 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:06.309683    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp"]
Jul 14 01:30:06.311791 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:30:06.311761    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332b0bac_8489_4d8e_9680_853333551579.slice/crio-89dc8538e9ceddbd704093d7ad5501fcfa115b08c5a981e69f6b85ecf4704bfc WatchSource:0}: Error finding container 89dc8538e9ceddbd704093d7ad5501fcfa115b08c5a981e69f6b85ecf4704bfc: Status 404 returned error can't find the container with id 89dc8538e9ceddbd704093d7ad5501fcfa115b08c5a981e69f6b85ecf4704bfc
Jul 14 01:30:06.481695 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:06.481659    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp" event={"ID":"332b0bac-8489-4d8e-9680-853333551579","Type":"ContainerStarted","Data":"89dc8538e9ceddbd704093d7ad5501fcfa115b08c5a981e69f6b85ecf4704bfc"}
Jul 14 01:30:06.980314 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:06.975701    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01c04727-0997-4fbc-8d74-c13dfb40c3b6" path="/var/lib/kubelet/pods/01c04727-0997-4fbc-8d74-c13dfb40c3b6/volumes"
Jul 14 01:30:07.489212 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:07.489172    2567 generic.go:356] "Generic (PLEG): container finished" podID="332b0bac-8489-4d8e-9680-853333551579" containerID="5fd3852128b1de4fb65b338257a9a40276c9047b31908e7e61d8835f983ecd7c" exitCode=7
Jul 14 01:30:07.489403 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:07.489250    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp" event={"ID":"332b0bac-8489-4d8e-9680-853333551579","Type":"ContainerDied","Data":"5fd3852128b1de4fb65b338257a9a40276c9047b31908e7e61d8835f983ecd7c"}
Jul 14 01:30:07.489452 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:07.489421    2567 scope.go:117] "RemoveContainer" containerID="5fd3852128b1de4fb65b338257a9a40276c9047b31908e7e61d8835f983ecd7c"
Jul 14 01:30:08.496469 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:08.496432    2567 generic.go:356] "Generic (PLEG): container finished" podID="332b0bac-8489-4d8e-9680-853333551579" containerID="edbc76b545bd6e087b0b1c2a18d1a17b26e293d4fdb494bc96de13156782f74c" exitCode=7
Jul 14 01:30:08.497008 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:08.496518    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp" event={"ID":"332b0bac-8489-4d8e-9680-853333551579","Type":"ContainerDied","Data":"edbc76b545bd6e087b0b1c2a18d1a17b26e293d4fdb494bc96de13156782f74c"}
Jul 14 01:30:08.497008 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:08.496579    2567 scope.go:117] "RemoveContainer" containerID="5fd3852128b1de4fb65b338257a9a40276c9047b31908e7e61d8835f983ecd7c"
Jul 14 01:30:08.497008 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:08.496823    2567 scope.go:117] "RemoveContainer" containerID="edbc76b545bd6e087b0b1c2a18d1a17b26e293d4fdb494bc96de13156782f74c"
Jul 14 01:30:08.497175 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:30:08.497085    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-7149234b-vskwp_odh-ai-gateway-infra(332b0bac-8489-4d8e-9680-853333551579)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp" podUID="332b0bac-8489-4d8e-9680-853333551579"
Jul 14 01:30:09.481797 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:09.481760    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:30:09.505387 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:09.505363    2567 scope.go:117] "RemoveContainer" containerID="edbc76b545bd6e087b0b1c2a18d1a17b26e293d4fdb494bc96de13156782f74c"
Jul 14 01:30:09.505767 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:30:09.505548    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-7149234b-vskwp_odh-ai-gateway-infra(332b0bac-8489-4d8e-9680-853333551579)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp" podUID="332b0bac-8489-4d8e-9680-853333551579"
Jul 14 01:30:16.545775 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:16.545740    2567 generic.go:356] "Generic (PLEG): container finished" podID="03882f99-5bbb-4ec8-9cfc-febddbb71b70" containerID="c60a025334d678fad09a800ccc345fa79355ac8d6a15ec79f67d3b50354604d0" exitCode=0
Jul 14 01:30:16.546219 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:16.545809    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm" event={"ID":"03882f99-5bbb-4ec8-9cfc-febddbb71b70","Type":"ContainerDied","Data":"c60a025334d678fad09a800ccc345fa79355ac8d6a15ec79f67d3b50354604d0"}
Jul 14 01:30:16.546261 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:16.546225    2567 scope.go:117] "RemoveContainer" containerID="c60a025334d678fad09a800ccc345fa79355ac8d6a15ec79f67d3b50354604d0"
Jul 14 01:30:16.547235 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:16.547209    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:30:16.547316 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:16.547302    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:30:16.547366 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:16.547337    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:30:17.457207 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:17.457159    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:17.557353 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:17.557307    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm" event={"ID":"03882f99-5bbb-4ec8-9cfc-febddbb71b70","Type":"ContainerStarted","Data":"dec3cb9d24cb9463f28c16bc3a1d0b2cf1a553b88a7105e975a40e39adf239a8"}
Jul 14 01:30:18.378579 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:18.378540    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:18.379839 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:18.379800    2567 patch_prober.go:28] interesting pod/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 14 01:30:18.379991 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:18.379863    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm" podUID="03882f99-5bbb-4ec8-9cfc-febddbb71b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 14 01:30:19.377827 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:19.377785    2567 patch_prober.go:28] interesting pod/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 14 01:30:19.378281 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:19.377861    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm" podUID="03882f99-5bbb-4ec8-9cfc-febddbb71b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 14 01:30:20.377944 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:20.377906    2567 patch_prober.go:28] interesting pod/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 14 01:30:20.378389 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:20.377997    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm" podUID="03882f99-5bbb-4ec8-9cfc-febddbb71b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 14 01:30:20.971370 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:20.971331    2567 scope.go:117] "RemoveContainer" containerID="edbc76b545bd6e087b0b1c2a18d1a17b26e293d4fdb494bc96de13156782f74c"
Jul 14 01:30:21.377735 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:21.377696    2567 patch_prober.go:28] interesting pod/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 14 01:30:21.377915 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:21.377752    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm" podUID="03882f99-5bbb-4ec8-9cfc-febddbb71b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 14 01:30:21.581772 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:21.581682    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp" event={"ID":"332b0bac-8489-4d8e-9680-853333551579","Type":"ContainerStarted","Data":"57b9ef52c313380bc62d64ce27322ed5c6e63db29ecbfcadc1584184bb6925eb"}
Jul 14 01:30:21.596904 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:21.596848    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp" podStartSLOduration=16.596831398 podStartE2EDuration="16.596831398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:30:21.594566098 +0000 UTC m=+1851.158975153" watchObservedRunningTime="2026-07-14 01:30:21.596831398 +0000 UTC m=+1851.161240516"
Jul 14 01:30:22.381940 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:22.381897    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:22.382306 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:22.382284    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:22.382869 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:22.382846    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:30:22.589118 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:22.589086    2567 generic.go:356] "Generic (PLEG): container finished" podID="332b0bac-8489-4d8e-9680-853333551579" containerID="57b9ef52c313380bc62d64ce27322ed5c6e63db29ecbfcadc1584184bb6925eb" exitCode=0
Jul 14 01:30:22.589527 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:22.589153    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp" event={"ID":"332b0bac-8489-4d8e-9680-853333551579","Type":"ContainerDied","Data":"57b9ef52c313380bc62d64ce27322ed5c6e63db29ecbfcadc1584184bb6925eb"}
Jul 14 01:30:22.589527 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:22.589211    2567 scope.go:117] "RemoveContainer" containerID="edbc76b545bd6e087b0b1c2a18d1a17b26e293d4fdb494bc96de13156782f74c"
Jul 14 01:30:22.611375 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:22.611347    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp"]
Jul 14 01:30:23.738097 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:23.738069    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp"
Jul 14 01:30:23.784584 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:23.784549    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/332b0bac-8489-4d8e-9680-853333551579-maas-api-service-ca\") pod \"332b0bac-8489-4d8e-9680-853333551579\" (UID: \"332b0bac-8489-4d8e-9680-853333551579\") "
Jul 14 01:30:23.784908 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:23.784883    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/332b0bac-8489-4d8e-9680-853333551579-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "332b0bac-8489-4d8e-9680-853333551579" (UID: "332b0bac-8489-4d8e-9680-853333551579"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:30:23.886011 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:23.885948    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/332b0bac-8489-4d8e-9680-853333551579-maas-api-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:30:24.603155 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:24.603128    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp"
Jul 14 01:30:24.603334 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:24.603133    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp" event={"ID":"332b0bac-8489-4d8e-9680-853333551579","Type":"ContainerDied","Data":"89dc8538e9ceddbd704093d7ad5501fcfa115b08c5a981e69f6b85ecf4704bfc"}
Jul 14 01:30:24.603334 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:24.603261    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89dc8538e9ceddbd704093d7ad5501fcfa115b08c5a981e69f6b85ecf4704bfc"
Jul 14 01:30:24.628309 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:24.628274    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp"]
Jul 14 01:30:24.633394 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:24.633365    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-vskwp"]
Jul 14 01:30:24.974864 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:24.974827    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="332b0bac-8489-4d8e-9680-853333551579" path="/var/lib/kubelet/pods/332b0bac-8489-4d8e-9680-853333551579/volumes"
Jul 14 01:30:55.888766 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:55.888680    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx"]
Jul 14 01:30:55.889211 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:55.889029    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="332b0bac-8489-4d8e-9680-853333551579" containerName="revoke-keys"
Jul 14 01:30:55.889211 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:55.889040    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="332b0bac-8489-4d8e-9680-853333551579" containerName="revoke-keys"
Jul 14 01:30:55.889211 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:55.889055    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="332b0bac-8489-4d8e-9680-853333551579" containerName="revoke-keys"
Jul 14 01:30:55.889211 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:55.889060    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="332b0bac-8489-4d8e-9680-853333551579" containerName="revoke-keys"
Jul 14 01:30:55.889211 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:55.889071    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="332b0bac-8489-4d8e-9680-853333551579" containerName="revoke-keys"
Jul 14 01:30:55.889211 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:55.889080    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="332b0bac-8489-4d8e-9680-853333551579" containerName="revoke-keys"
Jul 14 01:30:55.889211 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:55.889145    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="332b0bac-8489-4d8e-9680-853333551579" containerName="revoke-keys"
Jul 14 01:30:55.889211 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:55.889155    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="332b0bac-8489-4d8e-9680-853333551579" containerName="revoke-keys"
Jul 14 01:30:55.889211 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:55.889162    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="01c04727-0997-4fbc-8d74-c13dfb40c3b6" containerName="istio-proxy"
Jul 14 01:30:55.892429 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:55.892409    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx"
Jul 14 01:30:55.894255 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:55.894229    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kh8h4\""
Jul 14 01:30:55.894381 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:55.894255    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:30:55.901801 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:55.901772    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx"]
Jul 14 01:30:56.065727 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:56.065686    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e30ff65-06f2-4a44-be6a-648d86d0fa23-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx\" (UID: \"3e30ff65-06f2-4a44-be6a-648d86d0fa23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx"
Jul 14 01:30:56.166641 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:56.166544    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e30ff65-06f2-4a44-be6a-648d86d0fa23-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx\" (UID: \"3e30ff65-06f2-4a44-be6a-648d86d0fa23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx"
Jul 14 01:30:56.167271 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:56.167248    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e30ff65-06f2-4a44-be6a-648d86d0fa23-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx\" (UID: \"3e30ff65-06f2-4a44-be6a-648d86d0fa23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx"
Jul 14 01:30:56.203259 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:56.203209    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx"
Jul 14 01:30:56.342605 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:56.342577    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx"]
Jul 14 01:30:56.344653 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:30:56.344629    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e30ff65_06f2_4a44_be6a_648d86d0fa23.slice/crio-466d26158973d39a5b8c44a98739f7752fec8b497dc7e386435ab501f1002db4 WatchSource:0}: Error finding container 466d26158973d39a5b8c44a98739f7752fec8b497dc7e386435ab501f1002db4: Status 404 returned error can't find the container with id 466d26158973d39a5b8c44a98739f7752fec8b497dc7e386435ab501f1002db4
Jul 14 01:30:56.769612 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:56.769521    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx" event={"ID":"3e30ff65-06f2-4a44-be6a-648d86d0fa23","Type":"ContainerStarted","Data":"ad3ef354a0b42fc1b79b302120c82b5141069e251cabd01d7e46428d57db22bf"}
Jul 14 01:30:56.769612 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:56.769574    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx" event={"ID":"3e30ff65-06f2-4a44-be6a-648d86d0fa23","Type":"ContainerStarted","Data":"466d26158973d39a5b8c44a98739f7752fec8b497dc7e386435ab501f1002db4"}
Jul 14 01:30:56.785896 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:56.785839    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx" podStartSLOduration=1.785820747 podStartE2EDuration="1.785820747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:30:56.783372871 +0000 UTC m=+1886.347781926" watchObservedRunningTime="2026-07-14 01:30:56.785820747 +0000 UTC m=+1886.350229829"
Jul 14 01:30:57.776837 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:57.776800    2567 generic.go:356] "Generic (PLEG): container finished" podID="3e30ff65-06f2-4a44-be6a-648d86d0fa23" containerID="ad3ef354a0b42fc1b79b302120c82b5141069e251cabd01d7e46428d57db22bf" exitCode=0
Jul 14 01:30:57.777281 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:57.776892    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx" event={"ID":"3e30ff65-06f2-4a44-be6a-648d86d0fa23","Type":"ContainerDied","Data":"ad3ef354a0b42fc1b79b302120c82b5141069e251cabd01d7e46428d57db22bf"}
Jul 14 01:30:58.918908 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:58.918877    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx"
Jul 14 01:30:59.090979 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:59.090860    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e30ff65-06f2-4a44-be6a-648d86d0fa23-maas-api-service-ca\") pod \"3e30ff65-06f2-4a44-be6a-648d86d0fa23\" (UID: \"3e30ff65-06f2-4a44-be6a-648d86d0fa23\") "
Jul 14 01:30:59.091282 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:59.091254    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e30ff65-06f2-4a44-be6a-648d86d0fa23-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3e30ff65-06f2-4a44-be6a-648d86d0fa23" (UID: "3e30ff65-06f2-4a44-be6a-648d86d0fa23"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:30:59.192394 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:59.192358    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e30ff65-06f2-4a44-be6a-648d86d0fa23-maas-api-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:30:59.788727 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:59.788698    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx"
Jul 14 01:30:59.788908 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:59.788697    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx" event={"ID":"3e30ff65-06f2-4a44-be6a-648d86d0fa23","Type":"ContainerDied","Data":"466d26158973d39a5b8c44a98739f7752fec8b497dc7e386435ab501f1002db4"}
Jul 14 01:30:59.788908 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:30:59.788818    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="466d26158973d39a5b8c44a98739f7752fec8b497dc7e386435ab501f1002db4"
Jul 14 01:31:05.962106 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:05.962070    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"]
Jul 14 01:31:05.962606 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:05.962309    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm" podUID="03882f99-5bbb-4ec8-9cfc-febddbb71b70" containerName="istio-proxy" containerID="cri-o://dec3cb9d24cb9463f28c16bc3a1d0b2cf1a553b88a7105e975a40e39adf239a8" gracePeriod=30
Jul 14 01:31:05.986022 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:05.985987    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:06.158137 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:06.158103    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:06.409613 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:06.409576    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:07.384975 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:07.384917    2567 patch_prober.go:28] interesting pod/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 01:31:07.385364 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:07.385004    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm" podUID="03882f99-5bbb-4ec8-9cfc-febddbb71b70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 01:31:11.228746 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.228669    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:31:11.300111 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.300077    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-data\") pod \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") "
Jul 14 01:31:11.300309 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.300122    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-token\") pod \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") "
Jul 14 01:31:11.300309 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.300146    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istiod-ca-cert\") pod \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") "
Jul 14 01:31:11.300309 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.300175    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-envoy\") pod \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") "
Jul 14 01:31:11.300309 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.300208    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwzdr\" (UniqueName: \"kubernetes.io/projected/03882f99-5bbb-4ec8-9cfc-febddbb71b70-kube-api-access-pwzdr\") pod \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") "
Jul 14 01:31:11.300309 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.300240    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-workload-certs\") pod \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") "
Jul 14 01:31:11.300309 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.300302    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-credential-socket\") pod \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") "
Jul 14 01:31:11.300630 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.300351    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-workload-socket\") pod \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") "
Jul 14 01:31:11.300630 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.300413    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-podinfo\") pod \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\" (UID: \"03882f99-5bbb-4ec8-9cfc-febddbb71b70\") "
Jul 14 01:31:11.300746 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.300694    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "03882f99-5bbb-4ec8-9cfc-febddbb71b70" (UID: "03882f99-5bbb-4ec8-9cfc-febddbb71b70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:31:11.300894 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.300871    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "03882f99-5bbb-4ec8-9cfc-febddbb71b70" (UID: "03882f99-5bbb-4ec8-9cfc-febddbb71b70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:31:11.301250 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.301113    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-data" (OuterVolumeSpecName: "istio-data") pod "03882f99-5bbb-4ec8-9cfc-febddbb71b70" (UID: "03882f99-5bbb-4ec8-9cfc-febddbb71b70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:31:11.301250 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.301152    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "03882f99-5bbb-4ec8-9cfc-febddbb71b70" (UID: "03882f99-5bbb-4ec8-9cfc-febddbb71b70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:31:11.301401 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.301275    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "03882f99-5bbb-4ec8-9cfc-febddbb71b70" (UID: "03882f99-5bbb-4ec8-9cfc-febddbb71b70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:31:11.302746 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.302716    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03882f99-5bbb-4ec8-9cfc-febddbb71b70-kube-api-access-pwzdr" (OuterVolumeSpecName: "kube-api-access-pwzdr") pod "03882f99-5bbb-4ec8-9cfc-febddbb71b70" (UID: "03882f99-5bbb-4ec8-9cfc-febddbb71b70"). InnerVolumeSpecName "kube-api-access-pwzdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:31:11.302857 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.302789    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-token" (OuterVolumeSpecName: "istio-token") pod "03882f99-5bbb-4ec8-9cfc-febddbb71b70" (UID: "03882f99-5bbb-4ec8-9cfc-febddbb71b70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:31:11.303177 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.302981    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "03882f99-5bbb-4ec8-9cfc-febddbb71b70" (UID: "03882f99-5bbb-4ec8-9cfc-febddbb71b70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:31:11.303292 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.303267    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "03882f99-5bbb-4ec8-9cfc-febddbb71b70" (UID: "03882f99-5bbb-4ec8-9cfc-febddbb71b70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 01:31:11.401660 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.401627    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-workload-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:31:11.401660 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.401655    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-podinfo\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:31:11.401869 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.401670    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-data\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:31:11.401869 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.401680    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-token\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:31:11.401869 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.401688    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istiod-ca-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:31:11.401869 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.401695    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-istio-envoy\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:31:11.401869 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.401707    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwzdr\" (UniqueName: \"kubernetes.io/projected/03882f99-5bbb-4ec8-9cfc-febddbb71b70-kube-api-access-pwzdr\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:31:11.401869 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.401720    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-workload-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:31:11.401869 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.401732    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03882f99-5bbb-4ec8-9cfc-febddbb71b70-credential-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:31:11.746178 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.746144    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:11.847107 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.847074    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:11.853099 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.853060    2567 generic.go:356] "Generic (PLEG): container finished" podID="03882f99-5bbb-4ec8-9cfc-febddbb71b70" containerID="dec3cb9d24cb9463f28c16bc3a1d0b2cf1a553b88a7105e975a40e39adf239a8" exitCode=0
Jul 14 01:31:11.853298 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.853113    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"
Jul 14 01:31:11.853298 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.853154    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm" event={"ID":"03882f99-5bbb-4ec8-9cfc-febddbb71b70","Type":"ContainerDied","Data":"dec3cb9d24cb9463f28c16bc3a1d0b2cf1a553b88a7105e975a40e39adf239a8"}
Jul 14 01:31:11.853298 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.853246    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm" event={"ID":"03882f99-5bbb-4ec8-9cfc-febddbb71b70","Type":"ContainerDied","Data":"81e8dfea9fe6a1e248b8774a9a9d8860dce43c1bcfcbf54ebd7291699202e599"}
Jul 14 01:31:11.853298 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.853277    2567 scope.go:117] "RemoveContainer" containerID="dec3cb9d24cb9463f28c16bc3a1d0b2cf1a553b88a7105e975a40e39adf239a8"
Jul 14 01:31:11.863574 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.863551    2567 scope.go:117] "RemoveContainer" containerID="c60a025334d678fad09a800ccc345fa79355ac8d6a15ec79f67d3b50354604d0"
Jul 14 01:31:11.873057 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.873035    2567 scope.go:117] "RemoveContainer" containerID="dec3cb9d24cb9463f28c16bc3a1d0b2cf1a553b88a7105e975a40e39adf239a8"
Jul 14 01:31:11.873370 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:31:11.873351    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec3cb9d24cb9463f28c16bc3a1d0b2cf1a553b88a7105e975a40e39adf239a8\": container with ID starting with dec3cb9d24cb9463f28c16bc3a1d0b2cf1a553b88a7105e975a40e39adf239a8 not found: ID does not exist" containerID="dec3cb9d24cb9463f28c16bc3a1d0b2cf1a553b88a7105e975a40e39adf239a8"
Jul 14 01:31:11.873448 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.873378    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec3cb9d24cb9463f28c16bc3a1d0b2cf1a553b88a7105e975a40e39adf239a8"} err="failed to get container status \"dec3cb9d24cb9463f28c16bc3a1d0b2cf1a553b88a7105e975a40e39adf239a8\": rpc error: code = NotFound desc = could not find container \"dec3cb9d24cb9463f28c16bc3a1d0b2cf1a553b88a7105e975a40e39adf239a8\": container with ID starting with dec3cb9d24cb9463f28c16bc3a1d0b2cf1a553b88a7105e975a40e39adf239a8 not found: ID does not exist"
Jul 14 01:31:11.873448 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.873396    2567 scope.go:117] "RemoveContainer" containerID="c60a025334d678fad09a800ccc345fa79355ac8d6a15ec79f67d3b50354604d0"
Jul 14 01:31:11.873648 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:31:11.873632    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60a025334d678fad09a800ccc345fa79355ac8d6a15ec79f67d3b50354604d0\": container with ID starting with c60a025334d678fad09a800ccc345fa79355ac8d6a15ec79f67d3b50354604d0 not found: ID does not exist" containerID="c60a025334d678fad09a800ccc345fa79355ac8d6a15ec79f67d3b50354604d0"
Jul 14 01:31:11.873688 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.873651    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60a025334d678fad09a800ccc345fa79355ac8d6a15ec79f67d3b50354604d0"} err="failed to get container status \"c60a025334d678fad09a800ccc345fa79355ac8d6a15ec79f67d3b50354604d0\": rpc error: code = NotFound desc = could not find container \"c60a025334d678fad09a800ccc345fa79355ac8d6a15ec79f67d3b50354604d0\": container with ID starting with c60a025334d678fad09a800ccc345fa79355ac8d6a15ec79f67d3b50354604d0 not found: ID does not exist"
Jul 14 01:31:11.876856 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.876822    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"]
Jul 14 01:31:11.883529 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.883496    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7149234b-gw-openshift-default-67759cd74c-htczm"]
Jul 14 01:31:11.920454 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.920410    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"]
Jul 14 01:31:11.920750 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.920734    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03882f99-5bbb-4ec8-9cfc-febddbb71b70" containerName="istio-proxy"
Jul 14 01:31:11.920750 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.920749    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="03882f99-5bbb-4ec8-9cfc-febddbb71b70" containerName="istio-proxy"
Jul 14 01:31:11.920886 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.920772    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03882f99-5bbb-4ec8-9cfc-febddbb71b70" containerName="istio-proxy"
Jul 14 01:31:11.920886 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.920778    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="03882f99-5bbb-4ec8-9cfc-febddbb71b70" containerName="istio-proxy"
Jul 14 01:31:11.920886 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.920787    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e30ff65-06f2-4a44-be6a-648d86d0fa23" containerName="revoke-keys"
Jul 14 01:31:11.920886 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.920798    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e30ff65-06f2-4a44-be6a-648d86d0fa23" containerName="revoke-keys"
Jul 14 01:31:11.921095 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.920888    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e30ff65-06f2-4a44-be6a-648d86d0fa23" containerName="revoke-keys"
Jul 14 01:31:11.921095 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.920904    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="03882f99-5bbb-4ec8-9cfc-febddbb71b70" containerName="istio-proxy"
Jul 14 01:31:11.921095 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.920912    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="03882f99-5bbb-4ec8-9cfc-febddbb71b70" containerName="istio-proxy"
Jul 14 01:31:11.921095 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.920920    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="332b0bac-8489-4d8e-9680-853333551579" containerName="revoke-keys"
Jul 14 01:31:11.926086 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.926059    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:11.928121 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.928092    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0d59bf49-openshift-default-dockercfg-zbrr9\""
Jul 14 01:31:11.936777 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.936744    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"]
Jul 14 01:31:11.956273 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:11.956240    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:12.005834 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.005741    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-credential-socket\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.005834 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.005792    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rgbj\" (UniqueName: \"kubernetes.io/projected/fda4a690-1eda-48e5-9fed-94ab496872c4-kube-api-access-8rgbj\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.006121 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.005841    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-token\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.006121 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.005869    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda4a690-1eda-48e5-9fed-94ab496872c4-istiod-ca-cert\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.006121 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.006002    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-podinfo\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.006121 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.006049    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-data\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.006288 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.006133    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-workload-socket\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.006288 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.006168    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-workload-certs\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.006288 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.006196    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-envoy\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.107540 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.107499    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rgbj\" (UniqueName: \"kubernetes.io/projected/fda4a690-1eda-48e5-9fed-94ab496872c4-kube-api-access-8rgbj\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.107734 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.107554    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-token\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.107734 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.107572    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda4a690-1eda-48e5-9fed-94ab496872c4-istiod-ca-cert\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.107734 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.107592    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-podinfo\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.107734 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.107651    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-data\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.107734 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.107716    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-workload-socket\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.107734 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.107736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-workload-certs\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.108115 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.107760    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-envoy\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.108115 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.107797    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-credential-socket\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.108224 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.108165    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-data\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.108263 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.108224    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-workload-socket\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.108263 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.108253    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-credential-socket\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.108339 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.108311    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-workload-certs\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.108882 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.108855    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda4a690-1eda-48e5-9fed-94ab496872c4-istiod-ca-cert\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.110276 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.110254    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-envoy\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.110417 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.110398    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-podinfo\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.117562 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.117539    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-token\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.117822 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.117801    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rgbj\" (UniqueName: \"kubernetes.io/projected/fda4a690-1eda-48e5-9fed-94ab496872c4-kube-api-access-8rgbj\") pod \"e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") " pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.239782 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.239734    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:12.389135 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.389094    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"]
Jul 14 01:31:12.391111 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:31:12.391079    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda4a690_1eda_48e5_9fed_94ab496872c4.slice/crio-58a92295d4c86f500d1647bb06d67b43e2d87a021a684e27c1475ea626122b49 WatchSource:0}: Error finding container 58a92295d4c86f500d1647bb06d67b43e2d87a021a684e27c1475ea626122b49: Status 404 returned error can't find the container with id 58a92295d4c86f500d1647bb06d67b43e2d87a021a684e27c1475ea626122b49
Jul 14 01:31:12.393491 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.393457    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:31:12.393595 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.393528    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:31:12.393595 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.393563    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:31:12.394292 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.394270    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:12.467768 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.467723    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:12.861407 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.861370    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr" event={"ID":"fda4a690-1eda-48e5-9fed-94ab496872c4","Type":"ContainerStarted","Data":"39bbfa4a726ba084553c26712c0ff0d8d684340382dc8df0bd3c950412756cac"}
Jul 14 01:31:12.861622 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.861429    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr" event={"ID":"fda4a690-1eda-48e5-9fed-94ab496872c4","Type":"ContainerStarted","Data":"58a92295d4c86f500d1647bb06d67b43e2d87a021a684e27c1475ea626122b49"}
Jul 14 01:31:12.880870 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.880787    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr" podStartSLOduration=1.880761794 podStartE2EDuration="1.880761794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:31:12.878086623 +0000 UTC m=+1902.442495701" watchObservedRunningTime="2026-07-14 01:31:12.880761794 +0000 UTC m=+1902.445170851"
Jul 14 01:31:12.975930 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:12.975894    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03882f99-5bbb-4ec8-9cfc-febddbb71b70" path="/var/lib/kubelet/pods/03882f99-5bbb-4ec8-9cfc-febddbb71b70/volumes"
Jul 14 01:31:13.066258 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:13.066220    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:13.152935 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:13.152905    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:13.211031 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:13.210994    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:13.240613 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:13.240579    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:13.245586 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:13.245560    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:13.358776 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:13.358721    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:13.808615 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:13.808576    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:13.868019 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:13.867982    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:13.868917 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:13.868896    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:13.890439 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:13.890409    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:15.788356 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:15.788320    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:18.153595 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:18.153554    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd"]
Jul 14 01:31:18.157234 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:18.157213    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd"
Jul 14 01:31:18.159095 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:18.159069    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kh8h4\""
Jul 14 01:31:18.159241 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:18.159177    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:31:18.163244 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:18.163214    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd"]
Jul 14 01:31:18.264491 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:18.264447    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba0f211-41a3-4649-a41e-33058b68f3bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd\" (UID: \"9ba0f211-41a3-4649-a41e-33058b68f3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd"
Jul 14 01:31:18.366057 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:18.366017    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba0f211-41a3-4649-a41e-33058b68f3bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd\" (UID: \"9ba0f211-41a3-4649-a41e-33058b68f3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd"
Jul 14 01:31:18.366652 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:18.366631    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba0f211-41a3-4649-a41e-33058b68f3bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd\" (UID: \"9ba0f211-41a3-4649-a41e-33058b68f3bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd"
Jul 14 01:31:18.470096 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:18.470014    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd"
Jul 14 01:31:18.611882 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:18.611852    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd"]
Jul 14 01:31:18.613913 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:31:18.613888    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba0f211_41a3_4649_a41e_33058b68f3bc.slice/crio-d81eab4a8f608bc0b92af5b7d29c33ac4f61581236cd9f856486b11698172de1 WatchSource:0}: Error finding container d81eab4a8f608bc0b92af5b7d29c33ac4f61581236cd9f856486b11698172de1: Status 404 returned error can't find the container with id d81eab4a8f608bc0b92af5b7d29c33ac4f61581236cd9f856486b11698172de1
Jul 14 01:31:18.904926 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:18.904889    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd" event={"ID":"9ba0f211-41a3-4649-a41e-33058b68f3bc","Type":"ContainerStarted","Data":"45630e71608e5516d943331ad25278e0c30875f24654cae619a3024256f540df"}
Jul 14 01:31:18.905115 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:18.904942    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd" event={"ID":"9ba0f211-41a3-4649-a41e-33058b68f3bc","Type":"ContainerStarted","Data":"d81eab4a8f608bc0b92af5b7d29c33ac4f61581236cd9f856486b11698172de1"}
Jul 14 01:31:18.921334 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:18.921275    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd" podStartSLOduration=0.921258094 podStartE2EDuration="921.258094ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:31:18.918584729 +0000 UTC m=+1908.482993786" watchObservedRunningTime="2026-07-14 01:31:18.921258094 +0000 UTC m=+1908.485667149"
Jul 14 01:31:19.911630 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:19.911590    2567 generic.go:356] "Generic (PLEG): container finished" podID="9ba0f211-41a3-4649-a41e-33058b68f3bc" containerID="45630e71608e5516d943331ad25278e0c30875f24654cae619a3024256f540df" exitCode=7
Jul 14 01:31:19.912046 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:19.911639    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd" event={"ID":"9ba0f211-41a3-4649-a41e-33058b68f3bc","Type":"ContainerDied","Data":"45630e71608e5516d943331ad25278e0c30875f24654cae619a3024256f540df"}
Jul 14 01:31:19.912046 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:19.911870    2567 scope.go:117] "RemoveContainer" containerID="45630e71608e5516d943331ad25278e0c30875f24654cae619a3024256f540df"
Jul 14 01:31:20.918876 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:20.918780    2567 generic.go:356] "Generic (PLEG): container finished" podID="9ba0f211-41a3-4649-a41e-33058b68f3bc" containerID="5f4c1a1cf52a2a50875a2257df49c40f9ede00b4a18091af4696844723ff5676" exitCode=7
Jul 14 01:31:20.918876 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:20.918845    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd" event={"ID":"9ba0f211-41a3-4649-a41e-33058b68f3bc","Type":"ContainerDied","Data":"5f4c1a1cf52a2a50875a2257df49c40f9ede00b4a18091af4696844723ff5676"}
Jul 14 01:31:20.919403 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:20.918917    2567 scope.go:117] "RemoveContainer" containerID="45630e71608e5516d943331ad25278e0c30875f24654cae619a3024256f540df"
Jul 14 01:31:20.919403 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:20.919101    2567 scope.go:117] "RemoveContainer" containerID="5f4c1a1cf52a2a50875a2257df49c40f9ede00b4a18091af4696844723ff5676"
Jul 14 01:31:20.919403 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:31:20.919334    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd_odh-ai-gateway-infra(9ba0f211-41a3-4649-a41e-33058b68f3bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd" podUID="9ba0f211-41a3-4649-a41e-33058b68f3bc"
Jul 14 01:31:21.925522 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:21.925490    2567 scope.go:117] "RemoveContainer" containerID="5f4c1a1cf52a2a50875a2257df49c40f9ede00b4a18091af4696844723ff5676"
Jul 14 01:31:21.925906 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:31:21.925685    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd_odh-ai-gateway-infra(9ba0f211-41a3-4649-a41e-33058b68f3bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd" podUID="9ba0f211-41a3-4649-a41e-33058b68f3bc"
Jul 14 01:31:24.260664 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:24.260635    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:24.361140 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:24.361109    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:28.869752 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:28.869712    2567 patch_prober.go:28] interesting pod/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Jul 14 01:31:28.870221 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:28.869772    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr" podUID="fda4a690-1eda-48e5-9fed-94ab496872c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 14 01:31:28.975200 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:28.975115    2567 generic.go:356] "Generic (PLEG): container finished" podID="fda4a690-1eda-48e5-9fed-94ab496872c4" containerID="39bbfa4a726ba084553c26712c0ff0d8d684340382dc8df0bd3c950412756cac" exitCode=0
Jul 14 01:31:28.975358 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:28.975191    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr" event={"ID":"fda4a690-1eda-48e5-9fed-94ab496872c4","Type":"ContainerDied","Data":"39bbfa4a726ba084553c26712c0ff0d8d684340382dc8df0bd3c950412756cac"}
Jul 14 01:31:28.975587 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:28.975573    2567 scope.go:117] "RemoveContainer" containerID="39bbfa4a726ba084553c26712c0ff0d8d684340382dc8df0bd3c950412756cac"
Jul 14 01:31:28.976598 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:28.976569    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:31:28.976718 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:28.976656    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:31:28.976718 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:28.976704    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:31:29.985126 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:29.985091    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr" event={"ID":"fda4a690-1eda-48e5-9fed-94ab496872c4","Type":"ContainerStarted","Data":"75d1cf5ad5c49309d9cf41835aa8ebda95055d1f8ce4cef7b1e5247ec2e14448"}
Jul 14 01:31:30.240497 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:30.240385    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:30.240679 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:30.240529    2567 patch_prober.go:28] interesting pod/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Jul 14 01:31:30.240679 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:30.240576    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr" podUID="fda4a690-1eda-48e5-9fed-94ab496872c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 14 01:31:31.241060 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:31.241019    2567 patch_prober.go:28] interesting pod/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Jul 14 01:31:31.241639 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:31.241606    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr" podUID="fda4a690-1eda-48e5-9fed-94ab496872c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 14 01:31:32.241183 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:32.241133    2567 patch_prober.go:28] interesting pod/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Jul 14 01:31:32.241702 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:32.241208    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr" podUID="fda4a690-1eda-48e5-9fed-94ab496872c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 14 01:31:32.969165 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:32.969132    2567 scope.go:117] "RemoveContainer" containerID="5f4c1a1cf52a2a50875a2257df49c40f9ede00b4a18091af4696844723ff5676"
Jul 14 01:31:33.240723 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:33.240210    2567 patch_prober.go:28] interesting pod/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Jul 14 01:31:33.240723 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:33.240276    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr" podUID="fda4a690-1eda-48e5-9fed-94ab496872c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Jul 14 01:31:34.008501 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:34.008461    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd" event={"ID":"9ba0f211-41a3-4649-a41e-33058b68f3bc","Type":"ContainerStarted","Data":"4fffe011b6c293ddf121664b55038c0e2dd85a0f4dccbeda22bdb293e57904dd"}
Jul 14 01:31:34.244367 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:34.244285    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:34.244594 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:34.244573    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:34.245330 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:34.245309    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:31:34.639242 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:34.639206    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:34.740315 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:34.740282    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:35.015495 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:35.015397    2567 generic.go:356] "Generic (PLEG): container finished" podID="9ba0f211-41a3-4649-a41e-33058b68f3bc" containerID="4fffe011b6c293ddf121664b55038c0e2dd85a0f4dccbeda22bdb293e57904dd" exitCode=0
Jul 14 01:31:35.015495 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:35.015477    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd" event={"ID":"9ba0f211-41a3-4649-a41e-33058b68f3bc","Type":"ContainerDied","Data":"4fffe011b6c293ddf121664b55038c0e2dd85a0f4dccbeda22bdb293e57904dd"}
Jul 14 01:31:35.016081 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:35.015552    2567 scope.go:117] "RemoveContainer" containerID="5f4c1a1cf52a2a50875a2257df49c40f9ede00b4a18091af4696844723ff5676"
Jul 14 01:31:35.037492 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:35.037457    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd"]
Jul 14 01:31:36.162000 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:36.161947    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd"
Jul 14 01:31:36.226932 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:36.226893    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba0f211-41a3-4649-a41e-33058b68f3bc-maas-api-service-ca\") pod \"9ba0f211-41a3-4649-a41e-33058b68f3bc\" (UID: \"9ba0f211-41a3-4649-a41e-33058b68f3bc\") "
Jul 14 01:31:36.227299 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:36.227275    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba0f211-41a3-4649-a41e-33058b68f3bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ba0f211-41a3-4649-a41e-33058b68f3bc" (UID: "9ba0f211-41a3-4649-a41e-33058b68f3bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:31:36.328403 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:36.328305    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba0f211-41a3-4649-a41e-33058b68f3bc-maas-api-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:31:37.028772 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:37.028746    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd"
Jul 14 01:31:37.029009 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:37.028782    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd" event={"ID":"9ba0f211-41a3-4649-a41e-33058b68f3bc","Type":"ContainerDied","Data":"d81eab4a8f608bc0b92af5b7d29c33ac4f61581236cd9f856486b11698172de1"}
Jul 14 01:31:37.029009 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:37.028833    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d81eab4a8f608bc0b92af5b7d29c33ac4f61581236cd9f856486b11698172de1"
Jul 14 01:31:37.056173 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:37.056133    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd"]
Jul 14 01:31:37.061291 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:37.061262    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-dkwcd"]
Jul 14 01:31:38.975013 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:38.974976    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ba0f211-41a3-4649-a41e-33058b68f3bc" path="/var/lib/kubelet/pods/9ba0f211-41a3-4649-a41e-33058b68f3bc/volumes"
Jul 14 01:31:58.507459 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:58.507424    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:31:58.608440 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:31:58.608391    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:08.198040 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.198004    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4"]
Jul 14 01:32:08.198556 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.198365    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba0f211-41a3-4649-a41e-33058b68f3bc" containerName="revoke-keys"
Jul 14 01:32:08.198556 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.198376    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba0f211-41a3-4649-a41e-33058b68f3bc" containerName="revoke-keys"
Jul 14 01:32:08.198556 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.198403    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba0f211-41a3-4649-a41e-33058b68f3bc" containerName="revoke-keys"
Jul 14 01:32:08.198556 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.198409    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba0f211-41a3-4649-a41e-33058b68f3bc" containerName="revoke-keys"
Jul 14 01:32:08.198556 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.198465    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ba0f211-41a3-4649-a41e-33058b68f3bc" containerName="revoke-keys"
Jul 14 01:32:08.198556 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.198472    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ba0f211-41a3-4649-a41e-33058b68f3bc" containerName="revoke-keys"
Jul 14 01:32:08.201515 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.201498    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4"
Jul 14 01:32:08.203554 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.203531    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:32:08.203676 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.203531    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kh8h4\""
Jul 14 01:32:08.209764 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.209728    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4"]
Jul 14 01:32:08.211528 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.211505    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c11d7790-d915-4b9e-94d0-8255e5ba50a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4\" (UID: \"c11d7790-d915-4b9e-94d0-8255e5ba50a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4"
Jul 14 01:32:08.312391 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.312348    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c11d7790-d915-4b9e-94d0-8255e5ba50a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4\" (UID: \"c11d7790-d915-4b9e-94d0-8255e5ba50a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4"
Jul 14 01:32:08.313077 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.313054    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c11d7790-d915-4b9e-94d0-8255e5ba50a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4\" (UID: \"c11d7790-d915-4b9e-94d0-8255e5ba50a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4"
Jul 14 01:32:08.513614 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.513520    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4"
Jul 14 01:32:08.648830 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:08.648803    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4"]
Jul 14 01:32:08.650630 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:32:08.650601    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11d7790_d915_4b9e_94d0_8255e5ba50a8.slice/crio-5735add53e33f338237fab4fbef5ed07a6f994adb1e64b8a510b7a5f385234ce WatchSource:0}: Error finding container 5735add53e33f338237fab4fbef5ed07a6f994adb1e64b8a510b7a5f385234ce: Status 404 returned error can't find the container with id 5735add53e33f338237fab4fbef5ed07a6f994adb1e64b8a510b7a5f385234ce
Jul 14 01:32:09.197508 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:09.197460    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4" event={"ID":"c11d7790-d915-4b9e-94d0-8255e5ba50a8","Type":"ContainerStarted","Data":"d0847fbed3a247737d817882725e0e6649969770eafd2b1df523ab23dd53cf89"}
Jul 14 01:32:09.197718 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:09.197532    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4" event={"ID":"c11d7790-d915-4b9e-94d0-8255e5ba50a8","Type":"ContainerStarted","Data":"5735add53e33f338237fab4fbef5ed07a6f994adb1e64b8a510b7a5f385234ce"}
Jul 14 01:32:09.212466 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:09.212413    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4" podStartSLOduration=1.212395358 podStartE2EDuration="1.212395358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:32:09.211370627 +0000 UTC m=+1958.775779685" watchObservedRunningTime="2026-07-14 01:32:09.212395358 +0000 UTC m=+1958.776804410"
Jul 14 01:32:10.204075 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:10.204038    2567 generic.go:356] "Generic (PLEG): container finished" podID="c11d7790-d915-4b9e-94d0-8255e5ba50a8" containerID="d0847fbed3a247737d817882725e0e6649969770eafd2b1df523ab23dd53cf89" exitCode=0
Jul 14 01:32:10.204262 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:10.204130    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4" event={"ID":"c11d7790-d915-4b9e-94d0-8255e5ba50a8","Type":"ContainerDied","Data":"d0847fbed3a247737d817882725e0e6649969770eafd2b1df523ab23dd53cf89"}
Jul 14 01:32:11.344049 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:11.344018    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4"
Jul 14 01:32:11.436304 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:11.436263    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c11d7790-d915-4b9e-94d0-8255e5ba50a8-maas-api-service-ca\") pod \"c11d7790-d915-4b9e-94d0-8255e5ba50a8\" (UID: \"c11d7790-d915-4b9e-94d0-8255e5ba50a8\") "
Jul 14 01:32:11.436654 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:11.436630    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c11d7790-d915-4b9e-94d0-8255e5ba50a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c11d7790-d915-4b9e-94d0-8255e5ba50a8" (UID: "c11d7790-d915-4b9e-94d0-8255e5ba50a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:32:11.537488 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:11.537400    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c11d7790-d915-4b9e-94d0-8255e5ba50a8-maas-api-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:32:12.216814 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:12.216786    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4"
Jul 14 01:32:12.217018 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:12.216818    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4" event={"ID":"c11d7790-d915-4b9e-94d0-8255e5ba50a8","Type":"ContainerDied","Data":"5735add53e33f338237fab4fbef5ed07a6f994adb1e64b8a510b7a5f385234ce"}
Jul 14 01:32:12.217018 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:12.216865    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5735add53e33f338237fab4fbef5ed07a6f994adb1e64b8a510b7a5f385234ce"
Jul 14 01:32:18.826772 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:18.826739    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:23.282317 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:23.282237    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:28.795137 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:28.795102    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:28.854011 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:28.853971    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:34.311616 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.311581    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"]
Jul 14 01:32:34.312106 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.311850    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr" podUID="fda4a690-1eda-48e5-9fed-94ab496872c4" containerName="istio-proxy" containerID="cri-o://75d1cf5ad5c49309d9cf41835aa8ebda95055d1f8ce4cef7b1e5247ec2e14448" gracePeriod=30
Jul 14 01:32:34.336166 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.336123    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:34.618105 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.618067    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"]
Jul 14 01:32:34.618591 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.618575    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba0f211-41a3-4649-a41e-33058b68f3bc" containerName="revoke-keys"
Jul 14 01:32:34.618636 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.618595    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba0f211-41a3-4649-a41e-33058b68f3bc" containerName="revoke-keys"
Jul 14 01:32:34.618636 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.618625    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c11d7790-d915-4b9e-94d0-8255e5ba50a8" containerName="revoke-keys"
Jul 14 01:32:34.618706 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.618634    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c11d7790-d915-4b9e-94d0-8255e5ba50a8" containerName="revoke-keys"
Jul 14 01:32:34.618745 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.618710    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ba0f211-41a3-4649-a41e-33058b68f3bc" containerName="revoke-keys"
Jul 14 01:32:34.618745 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.618727    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c11d7790-d915-4b9e-94d0-8255e5ba50a8" containerName="revoke-keys"
Jul 14 01:32:34.622161 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.622138    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.624074 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.624046    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0e23d941-openshift-default-dockercfg-nt2j9\""
Jul 14 01:32:34.632623 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.632591    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"]
Jul 14 01:32:34.651207 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.651163    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:34.708279 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.708239    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-token\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.708279 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.708281    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bwck\" (UniqueName: \"kubernetes.io/projected/ca9eef0b-1407-4950-bf9b-54a478de4e91-kube-api-access-2bwck\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.708523 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.708302    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-envoy\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.708523 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.708323    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-credential-socket\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.708523 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.708402    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9eef0b-1407-4950-bf9b-54a478de4e91-istiod-ca-cert\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.708523 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.708436    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-workload-socket\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.708523 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.708457    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-podinfo\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.708523 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.708473    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-workload-certs\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.708523 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.708490    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-data\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.809098 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.809061    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-workload-socket\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.809098 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.809103    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-podinfo\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.809366 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.809122    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-workload-certs\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.809366 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.809236    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-data\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.809366 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.809317    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-token\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.809366 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.809354    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwck\" (UniqueName: \"kubernetes.io/projected/ca9eef0b-1407-4950-bf9b-54a478de4e91-kube-api-access-2bwck\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.809589 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.809381    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-envoy\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.809589 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.809418    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-credential-socket\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.809589 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.809495    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9eef0b-1407-4950-bf9b-54a478de4e91-istiod-ca-cert\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.809589 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.809517    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-workload-socket\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.809589 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.809576    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-workload-certs\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.809780 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.809624    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-data\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.809818 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.809783    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-credential-socket\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.810423 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.810398    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9eef0b-1407-4950-bf9b-54a478de4e91-istiod-ca-cert\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.811737 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.811719    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-envoy\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.812096 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.812076    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-podinfo\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.816853 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.816823    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-token\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.816985 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.816939    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bwck\" (UniqueName: \"kubernetes.io/projected/ca9eef0b-1407-4950-bf9b-54a478de4e91-kube-api-access-2bwck\") pod \"e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") " pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:34.933846 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:34.933746    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:35.076310 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:35.076275    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"]
Jul 14 01:32:35.077766 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:32:35.077739    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9eef0b_1407_4950_bf9b_54a478de4e91.slice/crio-144e004b8c09324b18351c057511f477e3fc69c13338e92731f63c7bec577b20 WatchSource:0}: Error finding container 144e004b8c09324b18351c057511f477e3fc69c13338e92731f63c7bec577b20: Status 404 returned error can't find the container with id 144e004b8c09324b18351c057511f477e3fc69c13338e92731f63c7bec577b20
Jul 14 01:32:35.079971 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:35.079920    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:32:35.080098 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:35.080082    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:32:35.080142 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:35.080119    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:32:35.341250 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:35.341151    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5" event={"ID":"ca9eef0b-1407-4950-bf9b-54a478de4e91","Type":"ContainerStarted","Data":"61751a36f85796069e425f3f3e08d8f5f639dc4fbb7cbef29da45dc9f8f1ca6d"}
Jul 14 01:32:35.341250 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:35.341209    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5" event={"ID":"ca9eef0b-1407-4950-bf9b-54a478de4e91","Type":"ContainerStarted","Data":"144e004b8c09324b18351c057511f477e3fc69c13338e92731f63c7bec577b20"}
Jul 14 01:32:35.359719 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:35.359660    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5" podStartSLOduration=1.359644541 podStartE2EDuration="1.359644541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:32:35.35718904 +0000 UTC m=+1984.921598095" watchObservedRunningTime="2026-07-14 01:32:35.359644541 +0000 UTC m=+1984.924053596"
Jul 14 01:32:35.744950 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:35.744910    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:35.825983 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:35.825933    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:35.886935 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:35.886890    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:35.933879 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:35.933835    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:36.390156 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:36.390121    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:36.496166 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:36.496127    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:36.856258 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:36.856160    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:36.939087 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:36.939052    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:37.352346 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:37.352313    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:37.353247 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:37.353224    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:39.578207 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.578132    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:32:39.644148 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.644114    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-workload-certs\") pod \"fda4a690-1eda-48e5-9fed-94ab496872c4\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") "
Jul 14 01:32:39.644148 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.644149    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-envoy\") pod \"fda4a690-1eda-48e5-9fed-94ab496872c4\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") "
Jul 14 01:32:39.644415 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.644199    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-workload-socket\") pod \"fda4a690-1eda-48e5-9fed-94ab496872c4\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") "
Jul 14 01:32:39.644415 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.644235    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda4a690-1eda-48e5-9fed-94ab496872c4-istiod-ca-cert\") pod \"fda4a690-1eda-48e5-9fed-94ab496872c4\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") "
Jul 14 01:32:39.644415 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.644264    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-data\") pod \"fda4a690-1eda-48e5-9fed-94ab496872c4\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") "
Jul 14 01:32:39.644415 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.644310    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-credential-socket\") pod \"fda4a690-1eda-48e5-9fed-94ab496872c4\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") "
Jul 14 01:32:39.644415 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.644333    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-podinfo\") pod \"fda4a690-1eda-48e5-9fed-94ab496872c4\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") "
Jul 14 01:32:39.644415 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.644364    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rgbj\" (UniqueName: \"kubernetes.io/projected/fda4a690-1eda-48e5-9fed-94ab496872c4-kube-api-access-8rgbj\") pod \"fda4a690-1eda-48e5-9fed-94ab496872c4\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") "
Jul 14 01:32:39.644415 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.644413    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-token\") pod \"fda4a690-1eda-48e5-9fed-94ab496872c4\" (UID: \"fda4a690-1eda-48e5-9fed-94ab496872c4\") "
Jul 14 01:32:39.644791 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.644447    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fda4a690-1eda-48e5-9fed-94ab496872c4" (UID: "fda4a690-1eda-48e5-9fed-94ab496872c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:32:39.644791 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.644692    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fda4a690-1eda-48e5-9fed-94ab496872c4" (UID: "fda4a690-1eda-48e5-9fed-94ab496872c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:32:39.644791 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.644711    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-workload-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:32:39.644791 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.644759    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda4a690-1eda-48e5-9fed-94ab496872c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fda4a690-1eda-48e5-9fed-94ab496872c4" (UID: "fda4a690-1eda-48e5-9fed-94ab496872c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:32:39.645145 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.645093    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fda4a690-1eda-48e5-9fed-94ab496872c4" (UID: "fda4a690-1eda-48e5-9fed-94ab496872c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:32:39.646035 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.645999    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "fda4a690-1eda-48e5-9fed-94ab496872c4" (UID: "fda4a690-1eda-48e5-9fed-94ab496872c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:32:39.647359 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.647336    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fda4a690-1eda-48e5-9fed-94ab496872c4" (UID: "fda4a690-1eda-48e5-9fed-94ab496872c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:32:39.647455 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.647379    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "fda4a690-1eda-48e5-9fed-94ab496872c4" (UID: "fda4a690-1eda-48e5-9fed-94ab496872c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:32:39.647627 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.647604    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fda4a690-1eda-48e5-9fed-94ab496872c4" (UID: "fda4a690-1eda-48e5-9fed-94ab496872c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 01:32:39.648008 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.647986    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda4a690-1eda-48e5-9fed-94ab496872c4-kube-api-access-8rgbj" (OuterVolumeSpecName: "kube-api-access-8rgbj") pod "fda4a690-1eda-48e5-9fed-94ab496872c4" (UID: "fda4a690-1eda-48e5-9fed-94ab496872c4"). InnerVolumeSpecName "kube-api-access-8rgbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:32:39.745478 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.745439    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-workload-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:32:39.745478 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.745473    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fda4a690-1eda-48e5-9fed-94ab496872c4-istiod-ca-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:32:39.745691 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.745483    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-data\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:32:39.745691 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.745500    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-credential-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:32:39.745691 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.745509    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-podinfo\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:32:39.745691 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.745517    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rgbj\" (UniqueName: \"kubernetes.io/projected/fda4a690-1eda-48e5-9fed-94ab496872c4-kube-api-access-8rgbj\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:32:39.745691 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.745526    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-token\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:32:39.745691 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:39.745534    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fda4a690-1eda-48e5-9fed-94ab496872c4-istio-envoy\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:32:40.160554 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.160517    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj"]
Jul 14 01:32:40.160871 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.160858    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda4a690-1eda-48e5-9fed-94ab496872c4" containerName="istio-proxy"
Jul 14 01:32:40.160919 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.160873    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda4a690-1eda-48e5-9fed-94ab496872c4" containerName="istio-proxy"
Jul 14 01:32:40.160919 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.160881    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda4a690-1eda-48e5-9fed-94ab496872c4" containerName="istio-proxy"
Jul 14 01:32:40.160919 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.160887    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda4a690-1eda-48e5-9fed-94ab496872c4" containerName="istio-proxy"
Jul 14 01:32:40.161064 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.160977    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fda4a690-1eda-48e5-9fed-94ab496872c4" containerName="istio-proxy"
Jul 14 01:32:40.164296 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.164275    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj"
Jul 14 01:32:40.166389 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.166366    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:32:40.166500 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.166422    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kh8h4\""
Jul 14 01:32:40.170402 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.170372    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj"]
Jul 14 01:32:40.248272 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.248231    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f513ae61-793d-4493-b770-0bd78411c4e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj\" (UID: \"f513ae61-793d-4493-b770-0bd78411c4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj"
Jul 14 01:32:40.349597 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.349560    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f513ae61-793d-4493-b770-0bd78411c4e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj\" (UID: \"f513ae61-793d-4493-b770-0bd78411c4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj"
Jul 14 01:32:40.350198 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.350178    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f513ae61-793d-4493-b770-0bd78411c4e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj\" (UID: \"f513ae61-793d-4493-b770-0bd78411c4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj"
Jul 14 01:32:40.369840 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.369801    2567 generic.go:356] "Generic (PLEG): container finished" podID="fda4a690-1eda-48e5-9fed-94ab496872c4" containerID="75d1cf5ad5c49309d9cf41835aa8ebda95055d1f8ce4cef7b1e5247ec2e14448" exitCode=0
Jul 14 01:32:40.370031 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.369848    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"
Jul 14 01:32:40.370031 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.369894    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr" event={"ID":"fda4a690-1eda-48e5-9fed-94ab496872c4","Type":"ContainerDied","Data":"75d1cf5ad5c49309d9cf41835aa8ebda95055d1f8ce4cef7b1e5247ec2e14448"}
Jul 14 01:32:40.370031 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.369968    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr" event={"ID":"fda4a690-1eda-48e5-9fed-94ab496872c4","Type":"ContainerDied","Data":"58a92295d4c86f500d1647bb06d67b43e2d87a021a684e27c1475ea626122b49"}
Jul 14 01:32:40.370031 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.369984    2567 scope.go:117] "RemoveContainer" containerID="75d1cf5ad5c49309d9cf41835aa8ebda95055d1f8ce4cef7b1e5247ec2e14448"
Jul 14 01:32:40.379814 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.379794    2567 scope.go:117] "RemoveContainer" containerID="39bbfa4a726ba084553c26712c0ff0d8d684340382dc8df0bd3c950412756cac"
Jul 14 01:32:40.388683 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.388668    2567 scope.go:117] "RemoveContainer" containerID="75d1cf5ad5c49309d9cf41835aa8ebda95055d1f8ce4cef7b1e5247ec2e14448"
Jul 14 01:32:40.389027 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:32:40.389002    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d1cf5ad5c49309d9cf41835aa8ebda95055d1f8ce4cef7b1e5247ec2e14448\": container with ID starting with 75d1cf5ad5c49309d9cf41835aa8ebda95055d1f8ce4cef7b1e5247ec2e14448 not found: ID does not exist" containerID="75d1cf5ad5c49309d9cf41835aa8ebda95055d1f8ce4cef7b1e5247ec2e14448"
Jul 14 01:32:40.389136 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.389038    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d1cf5ad5c49309d9cf41835aa8ebda95055d1f8ce4cef7b1e5247ec2e14448"} err="failed to get container status \"75d1cf5ad5c49309d9cf41835aa8ebda95055d1f8ce4cef7b1e5247ec2e14448\": rpc error: code = NotFound desc = could not find container \"75d1cf5ad5c49309d9cf41835aa8ebda95055d1f8ce4cef7b1e5247ec2e14448\": container with ID starting with 75d1cf5ad5c49309d9cf41835aa8ebda95055d1f8ce4cef7b1e5247ec2e14448 not found: ID does not exist"
Jul 14 01:32:40.389136 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.389061    2567 scope.go:117] "RemoveContainer" containerID="39bbfa4a726ba084553c26712c0ff0d8d684340382dc8df0bd3c950412756cac"
Jul 14 01:32:40.389330 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:32:40.389311    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39bbfa4a726ba084553c26712c0ff0d8d684340382dc8df0bd3c950412756cac\": container with ID starting with 39bbfa4a726ba084553c26712c0ff0d8d684340382dc8df0bd3c950412756cac not found: ID does not exist" containerID="39bbfa4a726ba084553c26712c0ff0d8d684340382dc8df0bd3c950412756cac"
Jul 14 01:32:40.389385 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.389340    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39bbfa4a726ba084553c26712c0ff0d8d684340382dc8df0bd3c950412756cac"} err="failed to get container status \"39bbfa4a726ba084553c26712c0ff0d8d684340382dc8df0bd3c950412756cac\": rpc error: code = NotFound desc = could not find container \"39bbfa4a726ba084553c26712c0ff0d8d684340382dc8df0bd3c950412756cac\": container with ID starting with 39bbfa4a726ba084553c26712c0ff0d8d684340382dc8df0bd3c950412756cac not found: ID does not exist"
Jul 14 01:32:40.390687 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.390664    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"]
Jul 14 01:32:40.397665 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.397629    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0d59bf49-openshift-default-cc7cd48b5-gw7vr"]
Jul 14 01:32:40.477200 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.477105    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj"
Jul 14 01:32:40.619317 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.619290    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj"]
Jul 14 01:32:40.621368 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:32:40.621332    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf513ae61_793d_4493_b770_0bd78411c4e9.slice/crio-67ce2d2b7c8f6ae85b5c8366d672386cc6b05f056d0fb94c9d02564feccec7b2 WatchSource:0}: Error finding container 67ce2d2b7c8f6ae85b5c8366d672386cc6b05f056d0fb94c9d02564feccec7b2: Status 404 returned error can't find the container with id 67ce2d2b7c8f6ae85b5c8366d672386cc6b05f056d0fb94c9d02564feccec7b2
Jul 14 01:32:40.786850 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.786816    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:40.887774 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.887741    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:40.975331 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:40.975287    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fda4a690-1eda-48e5-9fed-94ab496872c4" path="/var/lib/kubelet/pods/fda4a690-1eda-48e5-9fed-94ab496872c4/volumes"
Jul 14 01:32:41.378286 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:41.378249    2567 generic.go:356] "Generic (PLEG): container finished" podID="f513ae61-793d-4493-b770-0bd78411c4e9" containerID="98fd87835e6fbe72a9ba7571887a58e377fbcb56e70678c47e09e6b2cb0e17f0" exitCode=7
Jul 14 01:32:41.378473 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:41.378337    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj" event={"ID":"f513ae61-793d-4493-b770-0bd78411c4e9","Type":"ContainerDied","Data":"98fd87835e6fbe72a9ba7571887a58e377fbcb56e70678c47e09e6b2cb0e17f0"}
Jul 14 01:32:41.378473 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:41.378384    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj" event={"ID":"f513ae61-793d-4493-b770-0bd78411c4e9","Type":"ContainerStarted","Data":"67ce2d2b7c8f6ae85b5c8366d672386cc6b05f056d0fb94c9d02564feccec7b2"}
Jul 14 01:32:41.378574 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:41.378559    2567 scope.go:117] "RemoveContainer" containerID="98fd87835e6fbe72a9ba7571887a58e377fbcb56e70678c47e09e6b2cb0e17f0"
Jul 14 01:32:42.328593 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:42.328548    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:42.378574 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:42.378533    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:42.385082 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:42.385052    2567 generic.go:356] "Generic (PLEG): container finished" podID="f513ae61-793d-4493-b770-0bd78411c4e9" containerID="e77a00d745abcf398204389ab4f7ebd43c32f24510400949af2b0e24766a070b" exitCode=7
Jul 14 01:32:42.385222 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:42.385131    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj" event={"ID":"f513ae61-793d-4493-b770-0bd78411c4e9","Type":"ContainerDied","Data":"e77a00d745abcf398204389ab4f7ebd43c32f24510400949af2b0e24766a070b"}
Jul 14 01:32:42.385222 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:42.385192    2567 scope.go:117] "RemoveContainer" containerID="98fd87835e6fbe72a9ba7571887a58e377fbcb56e70678c47e09e6b2cb0e17f0"
Jul 14 01:32:42.385374 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:42.385359    2567 scope.go:117] "RemoveContainer" containerID="e77a00d745abcf398204389ab4f7ebd43c32f24510400949af2b0e24766a070b"
Jul 14 01:32:42.385624 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:32:42.385601    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj_odh-ai-gateway-infra(f513ae61-793d-4493-b770-0bd78411c4e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj" podUID="f513ae61-793d-4493-b770-0bd78411c4e9"
Jul 14 01:32:43.392575 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:43.392546    2567 scope.go:117] "RemoveContainer" containerID="e77a00d745abcf398204389ab4f7ebd43c32f24510400949af2b0e24766a070b"
Jul 14 01:32:43.392991 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:32:43.392763    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj_odh-ai-gateway-infra(f513ae61-793d-4493-b770-0bd78411c4e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj" podUID="f513ae61-793d-4493-b770-0bd78411c4e9"
Jul 14 01:32:44.178209 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:44.178175    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:32:48.424674 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:48.424617    2567 generic.go:356] "Generic (PLEG): container finished" podID="ca9eef0b-1407-4950-bf9b-54a478de4e91" containerID="61751a36f85796069e425f3f3e08d8f5f639dc4fbb7cbef29da45dc9f8f1ca6d" exitCode=0
Jul 14 01:32:48.424674 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:48.424674    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5" event={"ID":"ca9eef0b-1407-4950-bf9b-54a478de4e91","Type":"ContainerDied","Data":"61751a36f85796069e425f3f3e08d8f5f639dc4fbb7cbef29da45dc9f8f1ca6d"}
Jul 14 01:32:48.425284 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:48.425199    2567 scope.go:117] "RemoveContainer" containerID="61751a36f85796069e425f3f3e08d8f5f639dc4fbb7cbef29da45dc9f8f1ca6d"
Jul 14 01:32:48.426268 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:48.426235    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:32:48.426367 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:48.426306    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:32:48.426367 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:48.426345    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:32:49.431715 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:49.431681    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5" event={"ID":"ca9eef0b-1407-4950-bf9b-54a478de4e91","Type":"ContainerStarted","Data":"bff29dda22e094bb0fbc4b9aaeff55e03845d259ef8e719c087b0ac43d3a6d93"}
Jul 14 01:32:49.933909 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:49.933856    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:49.935543 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:49.935505    2567 patch_prober.go:28] interesting pod/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 14 01:32:49.935716 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:49.935593    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5" podUID="ca9eef0b-1407-4950-bf9b-54a478de4e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 14 01:32:50.934821 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:50.934782    2567 patch_prober.go:28] interesting pod/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 14 01:32:50.935460 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:50.935424    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5" podUID="ca9eef0b-1407-4950-bf9b-54a478de4e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 14 01:32:51.935088 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:51.935045    2567 patch_prober.go:28] interesting pod/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 14 01:32:51.935717 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:51.935679    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5" podUID="ca9eef0b-1407-4950-bf9b-54a478de4e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 14 01:32:52.353237 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:52.353130    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:52.934597 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:52.934552    2567 patch_prober.go:28] interesting pod/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 14 01:32:52.934928 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:52.934897    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5" podUID="ca9eef0b-1407-4950-bf9b-54a478de4e91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 14 01:32:53.938007 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:53.937977    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:53.938914 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:53.938894    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:32:58.969449 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:58.969414    2567 scope.go:117] "RemoveContainer" containerID="e77a00d745abcf398204389ab4f7ebd43c32f24510400949af2b0e24766a070b"
Jul 14 01:32:59.505377 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:59.505289    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj" event={"ID":"f513ae61-793d-4493-b770-0bd78411c4e9","Type":"ContainerStarted","Data":"e79675677ef34b2c571a36ff72eb3c12fa653e8840901c7eea0cb652e547e2a9"}
Jul 14 01:32:59.521734 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:32:59.521679    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj" podStartSLOduration=19.521664667 podStartE2EDuration="19.521664667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:32:59.520600302 +0000 UTC m=+2009.085009357" watchObservedRunningTime="2026-07-14 01:32:59.521664667 +0000 UTC m=+2009.086073721"
Jul 14 01:33:00.512684 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:00.512648    2567 generic.go:356] "Generic (PLEG): container finished" podID="f513ae61-793d-4493-b770-0bd78411c4e9" containerID="e79675677ef34b2c571a36ff72eb3c12fa653e8840901c7eea0cb652e547e2a9" exitCode=0
Jul 14 01:33:00.513123 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:00.512721    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj" event={"ID":"f513ae61-793d-4493-b770-0bd78411c4e9","Type":"ContainerDied","Data":"e79675677ef34b2c571a36ff72eb3c12fa653e8840901c7eea0cb652e547e2a9"}
Jul 14 01:33:00.513123 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:00.512784    2567 scope.go:117] "RemoveContainer" containerID="e77a00d745abcf398204389ab4f7ebd43c32f24510400949af2b0e24766a070b"
Jul 14 01:33:00.533651 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:00.533619    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj"]
Jul 14 01:33:01.656020 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:01.655994    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj"
Jul 14 01:33:01.724754 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:01.724712    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f513ae61-793d-4493-b770-0bd78411c4e9-maas-api-service-ca\") pod \"f513ae61-793d-4493-b770-0bd78411c4e9\" (UID: \"f513ae61-793d-4493-b770-0bd78411c4e9\") "
Jul 14 01:33:01.725092 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:01.725068    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f513ae61-793d-4493-b770-0bd78411c4e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f513ae61-793d-4493-b770-0bd78411c4e9" (UID: "f513ae61-793d-4493-b770-0bd78411c4e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:33:01.825631 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:01.825546    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f513ae61-793d-4493-b770-0bd78411c4e9-maas-api-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:33:02.525514 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:02.525483    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj"
Jul 14 01:33:02.525700 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:02.525518    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj" event={"ID":"f513ae61-793d-4493-b770-0bd78411c4e9","Type":"ContainerDied","Data":"67ce2d2b7c8f6ae85b5c8366d672386cc6b05f056d0fb94c9d02564feccec7b2"}
Jul 14 01:33:02.525700 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:02.525566    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67ce2d2b7c8f6ae85b5c8366d672386cc6b05f056d0fb94c9d02564feccec7b2"
Jul 14 01:33:02.551372 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:02.551335    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj"]
Jul 14 01:33:02.555025 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:02.554994    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-kqhqj"]
Jul 14 01:33:02.974504 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:02.974469    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f513ae61-793d-4493-b770-0bd78411c4e9" path="/var/lib/kubelet/pods/f513ae61-793d-4493-b770-0bd78411c4e9/volumes"
Jul 14 01:33:40.204185 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.204147    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5"]
Jul 14 01:33:40.204599 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.204503    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f513ae61-793d-4493-b770-0bd78411c4e9" containerName="revoke-keys"
Jul 14 01:33:40.204599 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.204514    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f513ae61-793d-4493-b770-0bd78411c4e9" containerName="revoke-keys"
Jul 14 01:33:40.204599 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.204523    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f513ae61-793d-4493-b770-0bd78411c4e9" containerName="revoke-keys"
Jul 14 01:33:40.204599 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.204528    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f513ae61-793d-4493-b770-0bd78411c4e9" containerName="revoke-keys"
Jul 14 01:33:40.204599 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.204536    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f513ae61-793d-4493-b770-0bd78411c4e9" containerName="revoke-keys"
Jul 14 01:33:40.204599 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.204542    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f513ae61-793d-4493-b770-0bd78411c4e9" containerName="revoke-keys"
Jul 14 01:33:40.204599 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.204602    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fda4a690-1eda-48e5-9fed-94ab496872c4" containerName="istio-proxy"
Jul 14 01:33:40.204817 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.204609    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f513ae61-793d-4493-b770-0bd78411c4e9" containerName="revoke-keys"
Jul 14 01:33:40.204817 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.204616    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f513ae61-793d-4493-b770-0bd78411c4e9" containerName="revoke-keys"
Jul 14 01:33:40.204817 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.204622    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f513ae61-793d-4493-b770-0bd78411c4e9" containerName="revoke-keys"
Jul 14 01:33:40.207680 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.207662    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5"
Jul 14 01:33:40.209921 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.209897    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kh8h4\""
Jul 14 01:33:40.210096 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.209898    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:33:40.214369 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.214334    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5"]
Jul 14 01:33:40.368850 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.368808    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52dfa3b9-1be0-403d-9f78-348ba9b191d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5\" (UID: \"52dfa3b9-1be0-403d-9f78-348ba9b191d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5"
Jul 14 01:33:40.470275 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.470160    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52dfa3b9-1be0-403d-9f78-348ba9b191d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5\" (UID: \"52dfa3b9-1be0-403d-9f78-348ba9b191d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5"
Jul 14 01:33:40.470872 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.470845    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52dfa3b9-1be0-403d-9f78-348ba9b191d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5\" (UID: \"52dfa3b9-1be0-403d-9f78-348ba9b191d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5"
Jul 14 01:33:40.520065 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.520028    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5"
Jul 14 01:33:40.653407 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.653379    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5"]
Jul 14 01:33:40.656088 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:33:40.656057    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52dfa3b9_1be0_403d_9f78_348ba9b191d6.slice/crio-827ac660930a880efdb2dd22b5599bf0e8d04bb526ebd1b6127917675bf944ac WatchSource:0}: Error finding container 827ac660930a880efdb2dd22b5599bf0e8d04bb526ebd1b6127917675bf944ac: Status 404 returned error can't find the container with id 827ac660930a880efdb2dd22b5599bf0e8d04bb526ebd1b6127917675bf944ac
Jul 14 01:33:40.735746 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:40.735656    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5" event={"ID":"52dfa3b9-1be0-403d-9f78-348ba9b191d6","Type":"ContainerStarted","Data":"827ac660930a880efdb2dd22b5599bf0e8d04bb526ebd1b6127917675bf944ac"}
Jul 14 01:33:41.743122 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:41.743086    2567 generic.go:356] "Generic (PLEG): container finished" podID="52dfa3b9-1be0-403d-9f78-348ba9b191d6" containerID="fcd73575cd4f9d58f840713aad71df0ea8175044065c847458d4ca2178bad0f1" exitCode=0
Jul 14 01:33:41.743517 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:41.743176    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5" event={"ID":"52dfa3b9-1be0-403d-9f78-348ba9b191d6","Type":"ContainerDied","Data":"fcd73575cd4f9d58f840713aad71df0ea8175044065c847458d4ca2178bad0f1"}
Jul 14 01:33:42.884565 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:42.884535    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5"
Jul 14 01:33:42.993186 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:42.993151    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52dfa3b9-1be0-403d-9f78-348ba9b191d6-maas-api-service-ca\") pod \"52dfa3b9-1be0-403d-9f78-348ba9b191d6\" (UID: \"52dfa3b9-1be0-403d-9f78-348ba9b191d6\") "
Jul 14 01:33:42.993519 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:42.993496    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52dfa3b9-1be0-403d-9f78-348ba9b191d6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52dfa3b9-1be0-403d-9f78-348ba9b191d6" (UID: "52dfa3b9-1be0-403d-9f78-348ba9b191d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:33:43.094364 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:43.094272    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52dfa3b9-1be0-403d-9f78-348ba9b191d6-maas-api-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:33:43.755757 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:43.755720    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5" event={"ID":"52dfa3b9-1be0-403d-9f78-348ba9b191d6","Type":"ContainerDied","Data":"827ac660930a880efdb2dd22b5599bf0e8d04bb526ebd1b6127917675bf944ac"}
Jul 14 01:33:43.755949 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:43.755771    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="827ac660930a880efdb2dd22b5599bf0e8d04bb526ebd1b6127917675bf944ac"
Jul 14 01:33:43.755949 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:43.755734    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5"
Jul 14 01:33:50.630844 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:50.630788    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:33:55.284746 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:33:55.284668    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:00.946265 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:00.946229    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:01.042721 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:01.042684    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:05.453253 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:05.453216    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"]
Jul 14 01:34:05.453777 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:05.453478    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5" podUID="ca9eef0b-1407-4950-bf9b-54a478de4e91" containerName="istio-proxy" containerID="cri-o://bff29dda22e094bb0fbc4b9aaeff55e03845d259ef8e719c087b0ac43d3a6d93" gracePeriod=30
Jul 14 01:34:05.474660 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:05.474631    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:05.619829 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:05.619790    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:06.089008 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:06.088949    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:06.188692 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:06.188652    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:06.289500 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:06.289469    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:07.790325 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:07.790288    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:08.940546 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:08.940509    2567 patch_prober.go:28] interesting pod/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 01:34:08.940925 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:08.940597    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5" podUID="ca9eef0b-1407-4950-bf9b-54a478de4e91" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 01:34:10.710636 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.710604    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:34:10.827756 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.827715    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-podinfo\") pod \"ca9eef0b-1407-4950-bf9b-54a478de4e91\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") "
Jul 14 01:34:10.827973 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.827772    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-workload-certs\") pod \"ca9eef0b-1407-4950-bf9b-54a478de4e91\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") "
Jul 14 01:34:10.827973 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.827801    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bwck\" (UniqueName: \"kubernetes.io/projected/ca9eef0b-1407-4950-bf9b-54a478de4e91-kube-api-access-2bwck\") pod \"ca9eef0b-1407-4950-bf9b-54a478de4e91\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") "
Jul 14 01:34:10.827973 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.827849    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-credential-socket\") pod \"ca9eef0b-1407-4950-bf9b-54a478de4e91\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") "
Jul 14 01:34:10.827973 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.827878    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-envoy\") pod \"ca9eef0b-1407-4950-bf9b-54a478de4e91\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") "
Jul 14 01:34:10.827973 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.827902    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-data\") pod \"ca9eef0b-1407-4950-bf9b-54a478de4e91\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") "
Jul 14 01:34:10.827973 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.827948    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-token\") pod \"ca9eef0b-1407-4950-bf9b-54a478de4e91\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") "
Jul 14 01:34:10.828288 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.827994    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9eef0b-1407-4950-bf9b-54a478de4e91-istiod-ca-cert\") pod \"ca9eef0b-1407-4950-bf9b-54a478de4e91\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") "
Jul 14 01:34:10.828288 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.828115    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-workload-socket\") pod \"ca9eef0b-1407-4950-bf9b-54a478de4e91\" (UID: \"ca9eef0b-1407-4950-bf9b-54a478de4e91\") "
Jul 14 01:34:10.828288 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.828195    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca9eef0b-1407-4950-bf9b-54a478de4e91" (UID: "ca9eef0b-1407-4950-bf9b-54a478de4e91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:34:10.828288 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.828209    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca9eef0b-1407-4950-bf9b-54a478de4e91" (UID: "ca9eef0b-1407-4950-bf9b-54a478de4e91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:34:10.828477 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.828437    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-credential-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:34:10.828477 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.828457    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-workload-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:34:10.828477 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.828455    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca9eef0b-1407-4950-bf9b-54a478de4e91" (UID: "ca9eef0b-1407-4950-bf9b-54a478de4e91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:34:10.828770 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.828749    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9eef0b-1407-4950-bf9b-54a478de4e91-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca9eef0b-1407-4950-bf9b-54a478de4e91" (UID: "ca9eef0b-1407-4950-bf9b-54a478de4e91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:34:10.829029 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.829001    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca9eef0b-1407-4950-bf9b-54a478de4e91" (UID: "ca9eef0b-1407-4950-bf9b-54a478de4e91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:34:10.830239 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.830212    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca9eef0b-1407-4950-bf9b-54a478de4e91" (UID: "ca9eef0b-1407-4950-bf9b-54a478de4e91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 01:34:10.830366 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.830344    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9eef0b-1407-4950-bf9b-54a478de4e91-kube-api-access-2bwck" (OuterVolumeSpecName: "kube-api-access-2bwck") pod "ca9eef0b-1407-4950-bf9b-54a478de4e91" (UID: "ca9eef0b-1407-4950-bf9b-54a478de4e91"). InnerVolumeSpecName "kube-api-access-2bwck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:34:10.830412 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.830349    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca9eef0b-1407-4950-bf9b-54a478de4e91" (UID: "ca9eef0b-1407-4950-bf9b-54a478de4e91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:34:10.830412 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.830353    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca9eef0b-1407-4950-bf9b-54a478de4e91" (UID: "ca9eef0b-1407-4950-bf9b-54a478de4e91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:34:10.921252 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.921210    2567 generic.go:356] "Generic (PLEG): container finished" podID="ca9eef0b-1407-4950-bf9b-54a478de4e91" containerID="bff29dda22e094bb0fbc4b9aaeff55e03845d259ef8e719c087b0ac43d3a6d93" exitCode=0
Jul 14 01:34:10.921432 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.921267    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"
Jul 14 01:34:10.921432 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.921289    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5" event={"ID":"ca9eef0b-1407-4950-bf9b-54a478de4e91","Type":"ContainerDied","Data":"bff29dda22e094bb0fbc4b9aaeff55e03845d259ef8e719c087b0ac43d3a6d93"}
Jul 14 01:34:10.921432 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.921345    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5" event={"ID":"ca9eef0b-1407-4950-bf9b-54a478de4e91","Type":"ContainerDied","Data":"144e004b8c09324b18351c057511f477e3fc69c13338e92731f63c7bec577b20"}
Jul 14 01:34:10.921432 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.921367    2567 scope.go:117] "RemoveContainer" containerID="bff29dda22e094bb0fbc4b9aaeff55e03845d259ef8e719c087b0ac43d3a6d93"
Jul 14 01:34:10.928986 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.928926    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-workload-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:34:10.928986 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.928976    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-podinfo\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:34:10.929138 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.928992    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bwck\" (UniqueName: \"kubernetes.io/projected/ca9eef0b-1407-4950-bf9b-54a478de4e91-kube-api-access-2bwck\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:34:10.929138 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.929005    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-envoy\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:34:10.929138 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.929017    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-data\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:34:10.929138 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.929028    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9eef0b-1407-4950-bf9b-54a478de4e91-istio-token\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:34:10.929138 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.929038    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9eef0b-1407-4950-bf9b-54a478de4e91-istiod-ca-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:34:10.931135 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.931113    2567 scope.go:117] "RemoveContainer" containerID="61751a36f85796069e425f3f3e08d8f5f639dc4fbb7cbef29da45dc9f8f1ca6d"
Jul 14 01:34:10.940162 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.940140    2567 scope.go:117] "RemoveContainer" containerID="bff29dda22e094bb0fbc4b9aaeff55e03845d259ef8e719c087b0ac43d3a6d93"
Jul 14 01:34:10.940491 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:34:10.940464    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff29dda22e094bb0fbc4b9aaeff55e03845d259ef8e719c087b0ac43d3a6d93\": container with ID starting with bff29dda22e094bb0fbc4b9aaeff55e03845d259ef8e719c087b0ac43d3a6d93 not found: ID does not exist" containerID="bff29dda22e094bb0fbc4b9aaeff55e03845d259ef8e719c087b0ac43d3a6d93"
Jul 14 01:34:10.940578 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.940500    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff29dda22e094bb0fbc4b9aaeff55e03845d259ef8e719c087b0ac43d3a6d93"} err="failed to get container status \"bff29dda22e094bb0fbc4b9aaeff55e03845d259ef8e719c087b0ac43d3a6d93\": rpc error: code = NotFound desc = could not find container \"bff29dda22e094bb0fbc4b9aaeff55e03845d259ef8e719c087b0ac43d3a6d93\": container with ID starting with bff29dda22e094bb0fbc4b9aaeff55e03845d259ef8e719c087b0ac43d3a6d93 not found: ID does not exist"
Jul 14 01:34:10.940578 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.940519    2567 scope.go:117] "RemoveContainer" containerID="61751a36f85796069e425f3f3e08d8f5f639dc4fbb7cbef29da45dc9f8f1ca6d"
Jul 14 01:34:10.940945 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:34:10.940877    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61751a36f85796069e425f3f3e08d8f5f639dc4fbb7cbef29da45dc9f8f1ca6d\": container with ID starting with 61751a36f85796069e425f3f3e08d8f5f639dc4fbb7cbef29da45dc9f8f1ca6d not found: ID does not exist" containerID="61751a36f85796069e425f3f3e08d8f5f639dc4fbb7cbef29da45dc9f8f1ca6d"
Jul 14 01:34:10.940945 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.940921    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61751a36f85796069e425f3f3e08d8f5f639dc4fbb7cbef29da45dc9f8f1ca6d"} err="failed to get container status \"61751a36f85796069e425f3f3e08d8f5f639dc4fbb7cbef29da45dc9f8f1ca6d\": rpc error: code = NotFound desc = could not find container \"61751a36f85796069e425f3f3e08d8f5f639dc4fbb7cbef29da45dc9f8f1ca6d\": container with ID starting with 61751a36f85796069e425f3f3e08d8f5f639dc4fbb7cbef29da45dc9f8f1ca6d not found: ID does not exist"
Jul 14 01:34:10.943361 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.943339    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"]
Jul 14 01:34:10.950144 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.950114    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0e23d941-openshift-default-bb4675467-8pvl5"]
Jul 14 01:34:10.974193 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:10.974153    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca9eef0b-1407-4950-bf9b-54a478de4e91" path="/var/lib/kubelet/pods/ca9eef0b-1407-4950-bf9b-54a478de4e91/volumes"
Jul 14 01:34:11.976825 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:11.976780    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:13.873007 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:13.872974    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:13.977938 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:13.977900    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:14.077822 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:14.077781    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:14.174333 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:14.174251    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:14.620186 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:14.620154    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:40.621426 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:40.621371    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:46.501141 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:46.501087    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:46.605941 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:46.605905    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:34:46.703221 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:34:46.703179    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:02.613754 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:02.613719    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:08.404697 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:08.404655    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:08.462579 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:08.462538    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:08.537908 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:08.537872    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:08.637563 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:08.637533    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:19.166764 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:19.166728    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:20.007326 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:20.007276    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:25.771677 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:25.771593    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:25.826614 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:25.826580    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:25.909949 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:25.909917    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:26.005078 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:26.005040    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:26.101806 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:26.101716    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:26.905163 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:26.905121    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:27.073470 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:27.073435    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:27.163901 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:27.163805    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:28.792395 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:28.792353    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:31.132196 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:31.132157    2567 scope.go:117] "RemoveContainer" containerID="e8947eeee93a99fce65035c283c39ebf441b84f1646caa954a7944be2ead1797"
Jul 14 01:35:34.618861 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:34.618824    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:34.719474 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:34.719444    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:35.284368 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:35.284329    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:41.209638 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.209599    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:41.307110 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.307081    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:41.452866 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.452828    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:41.738553 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.738519    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"]
Jul 14 01:35:41.738886 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.738873    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca9eef0b-1407-4950-bf9b-54a478de4e91" containerName="istio-proxy"
Jul 14 01:35:41.738931 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.738889    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca9eef0b-1407-4950-bf9b-54a478de4e91" containerName="istio-proxy"
Jul 14 01:35:41.738931 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.738911    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52dfa3b9-1be0-403d-9f78-348ba9b191d6" containerName="revoke-keys"
Jul 14 01:35:41.738931 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.738916    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="52dfa3b9-1be0-403d-9f78-348ba9b191d6" containerName="revoke-keys"
Jul 14 01:35:41.739051 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.738986    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca9eef0b-1407-4950-bf9b-54a478de4e91" containerName="istio-proxy"
Jul 14 01:35:41.739051 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.738994    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="52dfa3b9-1be0-403d-9f78-348ba9b191d6" containerName="revoke-keys"
Jul 14 01:35:41.741983 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.741946    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.743921 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.743899    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-db5a4a31-openshift-default-dockercfg-5tk4m\""
Jul 14 01:35:41.759911 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.759878    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"]
Jul 14 01:35:41.766838 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.766805    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:41.828142 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.828105    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-workload-socket\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.828344 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.828155    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-istio-envoy\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.828344 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.828193    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-istio-data\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.828344 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.828233    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c85fb4-36bf-4605-9974-208647612028-istio-podinfo\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.828344 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.828287    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-credential-socket\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.828344 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.828313    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-workload-certs\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.828344 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.828332    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c85fb4-36bf-4605-9974-208647612028-istiod-ca-cert\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.828686 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.828372    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d987n\" (UniqueName: \"kubernetes.io/projected/17c85fb4-36bf-4605-9974-208647612028-kube-api-access-d987n\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.828686 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.828399    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c85fb4-36bf-4605-9974-208647612028-istio-token\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.929547 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.929504    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-workload-socket\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.929738 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.929561    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-istio-envoy\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.929738 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.929600    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-istio-data\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.929738 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.929636    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c85fb4-36bf-4605-9974-208647612028-istio-podinfo\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.929738 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.929673    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-credential-socket\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.929738 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.929700    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-workload-certs\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.929738 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.929725    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c85fb4-36bf-4605-9974-208647612028-istiod-ca-cert\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.930115 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.929764    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d987n\" (UniqueName: \"kubernetes.io/projected/17c85fb4-36bf-4605-9974-208647612028-kube-api-access-d987n\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.930115 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.929792    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c85fb4-36bf-4605-9974-208647612028-istio-token\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.930115 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.930023    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-workload-socket\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.930267 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.930163    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-credential-socket\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.930267 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.930200    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-workload-certs\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.930586 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.930560    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c85fb4-36bf-4605-9974-208647612028-istiod-ca-cert\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.930670 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.930568    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-istio-data\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.932164 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.932144    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-istio-envoy\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.932382 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.932364    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c85fb4-36bf-4605-9974-208647612028-istio-podinfo\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.941792 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.941744    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c85fb4-36bf-4605-9974-208647612028-istio-token\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:41.942084 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:41.942065    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d987n\" (UniqueName: \"kubernetes.io/projected/17c85fb4-36bf-4605-9974-208647612028-kube-api-access-d987n\") pod \"e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") " pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:42.052732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:42.052629    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:42.205498 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:42.205471    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"]
Jul 14 01:35:42.207636 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:35:42.207610    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c85fb4_36bf_4605_9974_208647612028.slice/crio-86224e2ae76a801f4aaaab30c13ecb9183d3451f849d01349f5f6d1a17931fd2 WatchSource:0}: Error finding container 86224e2ae76a801f4aaaab30c13ecb9183d3451f849d01349f5f6d1a17931fd2: Status 404 returned error can't find the container with id 86224e2ae76a801f4aaaab30c13ecb9183d3451f849d01349f5f6d1a17931fd2
Jul 14 01:35:42.210244 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:42.210214    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:35:42.210541 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:42.210281    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:35:42.210541 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:42.210310    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:35:42.273193 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:42.273158    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:42.425658 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:42.425614    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk" event={"ID":"17c85fb4-36bf-4605-9974-208647612028","Type":"ContainerStarted","Data":"b029877f4e4730a52732cdc3b56c9311566874beeffc618b999c26e3699d1de9"}
Jul 14 01:35:42.425862 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:42.425672    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk" event={"ID":"17c85fb4-36bf-4605-9974-208647612028","Type":"ContainerStarted","Data":"86224e2ae76a801f4aaaab30c13ecb9183d3451f849d01349f5f6d1a17931fd2"}
Jul 14 01:35:42.445541 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:42.445485    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk" podStartSLOduration=1.445467073 podStartE2EDuration="1.445467073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:35:42.443187542 +0000 UTC m=+2172.007596597" watchObservedRunningTime="2026-07-14 01:35:42.445467073 +0000 UTC m=+2172.009876127"
Jul 14 01:35:43.053093 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:43.053055    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:43.058154 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:43.058125    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:43.431381 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:43.431350    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:43.432345 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:43.432317    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:35:44.602050 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:44.602014    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:44.673167 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:44.673133    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:44.737317 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:44.737286    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:45.038297 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:45.038262    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:45.400592 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:45.400559    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:45.502998 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:45.502970    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:49.135239 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:49.135201    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:49.878465 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:49.878432    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:49.944927 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:49.944896    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:50.346536 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:50.346454    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:55.935286 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:55.935248    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:55.989924 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:55.989886    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:35:56.292251 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:56.292167    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk"]
Jul 14 01:35:56.292510 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:56.292498    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca9eef0b-1407-4950-bf9b-54a478de4e91" containerName="istio-proxy"
Jul 14 01:35:56.292552 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:56.292511    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca9eef0b-1407-4950-bf9b-54a478de4e91" containerName="istio-proxy"
Jul 14 01:35:56.292587 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:56.292576    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca9eef0b-1407-4950-bf9b-54a478de4e91" containerName="istio-proxy"
Jul 14 01:35:56.296794 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:56.296774    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk"
Jul 14 01:35:56.299151 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:56.299123    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:35:56.299299 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:56.299123    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kh8h4\""
Jul 14 01:35:56.303407 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:56.303382    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk"]
Jul 14 01:35:56.353432 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:56.353388    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f9639a-6472-4942-b7e0-994bd0223824-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk\" (UID: \"b4f9639a-6472-4942-b7e0-994bd0223824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk"
Jul 14 01:35:56.454035 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:56.453988    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f9639a-6472-4942-b7e0-994bd0223824-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk\" (UID: \"b4f9639a-6472-4942-b7e0-994bd0223824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk"
Jul 14 01:35:56.454665 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:56.454645    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f9639a-6472-4942-b7e0-994bd0223824-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk\" (UID: \"b4f9639a-6472-4942-b7e0-994bd0223824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk"
Jul 14 01:35:56.610483 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:56.610396    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk"
Jul 14 01:35:56.752321 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:56.752287    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk"]
Jul 14 01:35:56.754689 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:35:56.754654    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f9639a_6472_4942_b7e0_994bd0223824.slice/crio-11be47cb8b39b8c0c252f7f726ba4faa246d278a58df8cf67a38511cb259bf21 WatchSource:0}: Error finding container 11be47cb8b39b8c0c252f7f726ba4faa246d278a58df8cf67a38511cb259bf21: Status 404 returned error can't find the container with id 11be47cb8b39b8c0c252f7f726ba4faa246d278a58df8cf67a38511cb259bf21
Jul 14 01:35:57.515154 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:57.515121    2567 generic.go:356] "Generic (PLEG): container finished" podID="b4f9639a-6472-4942-b7e0-994bd0223824" containerID="b0712294f9edef41ff1dbc4388d09fce325bb476a86a18b809b6fe09a2beefb2" exitCode=0
Jul 14 01:35:57.515526 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:57.515212    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk" event={"ID":"b4f9639a-6472-4942-b7e0-994bd0223824","Type":"ContainerDied","Data":"b0712294f9edef41ff1dbc4388d09fce325bb476a86a18b809b6fe09a2beefb2"}
Jul 14 01:35:57.515526 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:57.515269    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk" event={"ID":"b4f9639a-6472-4942-b7e0-994bd0223824","Type":"ContainerStarted","Data":"11be47cb8b39b8c0c252f7f726ba4faa246d278a58df8cf67a38511cb259bf21"}
Jul 14 01:35:58.433041 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:58.433001    2567 patch_prober.go:28] interesting pod/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 14 01:35:58.433249 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:58.433080    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk" podUID="17c85fb4-36bf-4605-9974-208647612028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 14 01:35:58.523117 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:58.523084    2567 generic.go:356] "Generic (PLEG): container finished" podID="17c85fb4-36bf-4605-9974-208647612028" containerID="b029877f4e4730a52732cdc3b56c9311566874beeffc618b999c26e3699d1de9" exitCode=0
Jul 14 01:35:58.523533 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:58.523158    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk" event={"ID":"17c85fb4-36bf-4605-9974-208647612028","Type":"ContainerDied","Data":"b029877f4e4730a52732cdc3b56c9311566874beeffc618b999c26e3699d1de9"}
Jul 14 01:35:58.523882 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:58.523594    2567 scope.go:117] "RemoveContainer" containerID="b029877f4e4730a52732cdc3b56c9311566874beeffc618b999c26e3699d1de9"
Jul 14 01:35:58.524726 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:58.524696    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:35:58.524819 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:58.524769    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:35:58.524819 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:58.524807    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:35:58.696127 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:58.696044    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk"
Jul 14 01:35:58.775977 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:58.775914    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f9639a-6472-4942-b7e0-994bd0223824-maas-api-service-ca\") pod \"b4f9639a-6472-4942-b7e0-994bd0223824\" (UID: \"b4f9639a-6472-4942-b7e0-994bd0223824\") "
Jul 14 01:35:58.776301 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:58.776276    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f9639a-6472-4942-b7e0-994bd0223824-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4f9639a-6472-4942-b7e0-994bd0223824" (UID: "b4f9639a-6472-4942-b7e0-994bd0223824"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:35:58.877154 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:58.877110    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f9639a-6472-4942-b7e0-994bd0223824-maas-api-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:35:59.531198 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:59.531161    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk" event={"ID":"17c85fb4-36bf-4605-9974-208647612028","Type":"ContainerStarted","Data":"30c722fd5cea53a4d6bc04cd2fe649339f0f0c7bee46e32b37619f9aed25546f"}
Jul 14 01:35:59.532458 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:59.532439    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk"
Jul 14 01:35:59.532558 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:59.532459    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk" event={"ID":"b4f9639a-6472-4942-b7e0-994bd0223824","Type":"ContainerDied","Data":"11be47cb8b39b8c0c252f7f726ba4faa246d278a58df8cf67a38511cb259bf21"}
Jul 14 01:35:59.532558 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:35:59.532496    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11be47cb8b39b8c0c252f7f726ba4faa246d278a58df8cf67a38511cb259bf21"
Jul 14 01:36:00.052945 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:00.052842    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:36:00.053619 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:00.053589    2567 patch_prober.go:28] interesting pod/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 14 01:36:00.053856 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:00.053831    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk" podUID="17c85fb4-36bf-4605-9974-208647612028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 14 01:36:01.053860 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:01.053816    2567 patch_prober.go:28] interesting pod/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 14 01:36:01.054376 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:01.053891    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk" podUID="17c85fb4-36bf-4605-9974-208647612028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 14 01:36:02.053646 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:02.053601    2567 patch_prober.go:28] interesting pod/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 14 01:36:02.054003 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:02.053947    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk" podUID="17c85fb4-36bf-4605-9974-208647612028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 14 01:36:03.054149 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:03.054009    2567 patch_prober.go:28] interesting pod/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 14 01:36:03.054149 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:03.054076    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk" podUID="17c85fb4-36bf-4605-9974-208647612028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 14 01:36:04.057066 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:04.057033    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:36:04.057535 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:04.057435    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:36:04.058077 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:04.058059    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:36:06.706182 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:06.706146    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:11.377785 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:11.377746    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:17.148514 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:17.148475    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:17.209192 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:17.209163    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:22.044255 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:22.044213    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"]
Jul 14 01:36:22.044799 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:22.044469    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk" podUID="17c85fb4-36bf-4605-9974-208647612028" containerName="istio-proxy" containerID="cri-o://30c722fd5cea53a4d6bc04cd2fe649339f0f0c7bee46e32b37619f9aed25546f" gracePeriod=30
Jul 14 01:36:22.069047 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:22.068172    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:22.308108 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:22.308017    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:22.438296 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:22.438265    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:22.538026 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:22.537986    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:22.672861 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:22.672824    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:25.109417 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:25.109379    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:25.209275 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:25.209241    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:27.305567 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.305494    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:36:27.326097 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.326064    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-credential-socket\") pod \"17c85fb4-36bf-4605-9974-208647612028\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") "
Jul 14 01:36:27.326273 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.326116    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-workload-socket\") pod \"17c85fb4-36bf-4605-9974-208647612028\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") "
Jul 14 01:36:27.326273 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.326156    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-workload-certs\") pod \"17c85fb4-36bf-4605-9974-208647612028\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") "
Jul 14 01:36:27.326273 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.326182    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c85fb4-36bf-4605-9974-208647612028-istio-token\") pod \"17c85fb4-36bf-4605-9974-208647612028\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") "
Jul 14 01:36:27.326273 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.326226    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c85fb4-36bf-4605-9974-208647612028-istio-podinfo\") pod \"17c85fb4-36bf-4605-9974-208647612028\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") "
Jul 14 01:36:27.326273 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.326255    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-istio-envoy\") pod \"17c85fb4-36bf-4605-9974-208647612028\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") "
Jul 14 01:36:27.326542 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.326278    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-istio-data\") pod \"17c85fb4-36bf-4605-9974-208647612028\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") "
Jul 14 01:36:27.326542 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.326306    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d987n\" (UniqueName: \"kubernetes.io/projected/17c85fb4-36bf-4605-9974-208647612028-kube-api-access-d987n\") pod \"17c85fb4-36bf-4605-9974-208647612028\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") "
Jul 14 01:36:27.326542 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.326345    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c85fb4-36bf-4605-9974-208647612028-istiod-ca-cert\") pod \"17c85fb4-36bf-4605-9974-208647612028\" (UID: \"17c85fb4-36bf-4605-9974-208647612028\") "
Jul 14 01:36:27.326542 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.326378    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17c85fb4-36bf-4605-9974-208647612028" (UID: "17c85fb4-36bf-4605-9974-208647612028"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:36:27.326542 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.326488    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17c85fb4-36bf-4605-9974-208647612028" (UID: "17c85fb4-36bf-4605-9974-208647612028"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:36:27.326799 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.326626    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-credential-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:36:27.326799 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.326646    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-workload-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:36:27.326799 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.326784    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17c85fb4-36bf-4605-9974-208647612028" (UID: "17c85fb4-36bf-4605-9974-208647612028"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:36:27.326975 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.326798    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c85fb4-36bf-4605-9974-208647612028-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17c85fb4-36bf-4605-9974-208647612028" (UID: "17c85fb4-36bf-4605-9974-208647612028"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:36:27.327413 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.327373    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-istio-data" (OuterVolumeSpecName: "istio-data") pod "17c85fb4-36bf-4605-9974-208647612028" (UID: "17c85fb4-36bf-4605-9974-208647612028"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:36:27.329436 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.329408    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c85fb4-36bf-4605-9974-208647612028-kube-api-access-d987n" (OuterVolumeSpecName: "kube-api-access-d987n") pod "17c85fb4-36bf-4605-9974-208647612028" (UID: "17c85fb4-36bf-4605-9974-208647612028"). InnerVolumeSpecName "kube-api-access-d987n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:36:27.329436 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.329412    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17c85fb4-36bf-4605-9974-208647612028" (UID: "17c85fb4-36bf-4605-9974-208647612028"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:36:27.329579 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.329479    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c85fb4-36bf-4605-9974-208647612028-istio-token" (OuterVolumeSpecName: "istio-token") pod "17c85fb4-36bf-4605-9974-208647612028" (UID: "17c85fb4-36bf-4605-9974-208647612028"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:36:27.329579 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.329526    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17c85fb4-36bf-4605-9974-208647612028-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17c85fb4-36bf-4605-9974-208647612028" (UID: "17c85fb4-36bf-4605-9974-208647612028"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 01:36:27.427924 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.427885    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c85fb4-36bf-4605-9974-208647612028-istio-token\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:36:27.427924 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.427917    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c85fb4-36bf-4605-9974-208647612028-istio-podinfo\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:36:27.427924 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.427928    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-istio-envoy\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:36:27.427924 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.427937    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-istio-data\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:36:27.428222 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.427946    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d987n\" (UniqueName: \"kubernetes.io/projected/17c85fb4-36bf-4605-9974-208647612028-kube-api-access-d987n\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:36:27.428222 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.427977    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c85fb4-36bf-4605-9974-208647612028-istiod-ca-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:36:27.428222 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.427985    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c85fb4-36bf-4605-9974-208647612028-workload-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:36:27.696900 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.696866    2567 generic.go:356] "Generic (PLEG): container finished" podID="17c85fb4-36bf-4605-9974-208647612028" containerID="30c722fd5cea53a4d6bc04cd2fe649339f0f0c7bee46e32b37619f9aed25546f" exitCode=0
Jul 14 01:36:27.697107 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.696914    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"
Jul 14 01:36:27.697107 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.696989    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk" event={"ID":"17c85fb4-36bf-4605-9974-208647612028","Type":"ContainerDied","Data":"30c722fd5cea53a4d6bc04cd2fe649339f0f0c7bee46e32b37619f9aed25546f"}
Jul 14 01:36:27.697107 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.697048    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk" event={"ID":"17c85fb4-36bf-4605-9974-208647612028","Type":"ContainerDied","Data":"86224e2ae76a801f4aaaab30c13ecb9183d3451f849d01349f5f6d1a17931fd2"}
Jul 14 01:36:27.697107 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.697077    2567 scope.go:117] "RemoveContainer" containerID="30c722fd5cea53a4d6bc04cd2fe649339f0f0c7bee46e32b37619f9aed25546f"
Jul 14 01:36:27.710741 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.710711    2567 scope.go:117] "RemoveContainer" containerID="b029877f4e4730a52732cdc3b56c9311566874beeffc618b999c26e3699d1de9"
Jul 14 01:36:27.721662 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.721632    2567 scope.go:117] "RemoveContainer" containerID="30c722fd5cea53a4d6bc04cd2fe649339f0f0c7bee46e32b37619f9aed25546f"
Jul 14 01:36:27.722036 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:36:27.722011    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c722fd5cea53a4d6bc04cd2fe649339f0f0c7bee46e32b37619f9aed25546f\": container with ID starting with 30c722fd5cea53a4d6bc04cd2fe649339f0f0c7bee46e32b37619f9aed25546f not found: ID does not exist" containerID="30c722fd5cea53a4d6bc04cd2fe649339f0f0c7bee46e32b37619f9aed25546f"
Jul 14 01:36:27.722135 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.722045    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c722fd5cea53a4d6bc04cd2fe649339f0f0c7bee46e32b37619f9aed25546f"} err="failed to get container status \"30c722fd5cea53a4d6bc04cd2fe649339f0f0c7bee46e32b37619f9aed25546f\": rpc error: code = NotFound desc = could not find container \"30c722fd5cea53a4d6bc04cd2fe649339f0f0c7bee46e32b37619f9aed25546f\": container with ID starting with 30c722fd5cea53a4d6bc04cd2fe649339f0f0c7bee46e32b37619f9aed25546f not found: ID does not exist"
Jul 14 01:36:27.722135 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.722065    2567 scope.go:117] "RemoveContainer" containerID="b029877f4e4730a52732cdc3b56c9311566874beeffc618b999c26e3699d1de9"
Jul 14 01:36:27.722329 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:36:27.722309    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b029877f4e4730a52732cdc3b56c9311566874beeffc618b999c26e3699d1de9\": container with ID starting with b029877f4e4730a52732cdc3b56c9311566874beeffc618b999c26e3699d1de9 not found: ID does not exist" containerID="b029877f4e4730a52732cdc3b56c9311566874beeffc618b999c26e3699d1de9"
Jul 14 01:36:27.722393 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.722336    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b029877f4e4730a52732cdc3b56c9311566874beeffc618b999c26e3699d1de9"} err="failed to get container status \"b029877f4e4730a52732cdc3b56c9311566874beeffc618b999c26e3699d1de9\": rpc error: code = NotFound desc = could not find container \"b029877f4e4730a52732cdc3b56c9311566874beeffc618b999c26e3699d1de9\": container with ID starting with b029877f4e4730a52732cdc3b56c9311566874beeffc618b999c26e3699d1de9 not found: ID does not exist"
Jul 14 01:36:27.722654 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.722630    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"]
Jul 14 01:36:27.725760 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:27.725738    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-db5a4a31-openshift-default-5f55576f4c-5rcvk"]
Jul 14 01:36:28.454454 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:28.454413    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:28.563755 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:28.563720    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:28.654742 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:28.654710    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:28.819062 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:28.818940    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:28.974733 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:28.974683    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17c85fb4-36bf-4605-9974-208647612028" path="/var/lib/kubelet/pods/17c85fb4-36bf-4605-9974-208647612028/volumes"
Jul 14 01:36:31.154668 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:31.154629    2567 scope.go:117] "RemoveContainer" containerID="57b9ef52c313380bc62d64ce27322ed5c6e63db29ecbfcadc1584184bb6925eb"
Jul 14 01:36:34.490685 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:34.490649    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:34.592224 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:34.592182    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:41.313382 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.313334    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:41.614341 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.614254    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"]
Jul 14 01:36:41.614592 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.614580    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c85fb4-36bf-4605-9974-208647612028" containerName="istio-proxy"
Jul 14 01:36:41.614640 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.614594    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c85fb4-36bf-4605-9974-208647612028" containerName="istio-proxy"
Jul 14 01:36:41.614640 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.614608    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c85fb4-36bf-4605-9974-208647612028" containerName="istio-proxy"
Jul 14 01:36:41.614640 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.614615    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c85fb4-36bf-4605-9974-208647612028" containerName="istio-proxy"
Jul 14 01:36:41.614640 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.614628    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f9639a-6472-4942-b7e0-994bd0223824" containerName="revoke-keys"
Jul 14 01:36:41.614640 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.614634    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f9639a-6472-4942-b7e0-994bd0223824" containerName="revoke-keys"
Jul 14 01:36:41.614816 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.614692    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f9639a-6472-4942-b7e0-994bd0223824" containerName="revoke-keys"
Jul 14 01:36:41.614816 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.614701    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="17c85fb4-36bf-4605-9974-208647612028" containerName="istio-proxy"
Jul 14 01:36:41.614816 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.614708    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="17c85fb4-36bf-4605-9974-208647612028" containerName="istio-proxy"
Jul 14 01:36:41.617821 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.617803    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.619915 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.619894    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b02b4cdc-openshift-default-dockercfg-4p958\""
Jul 14 01:36:41.627215 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.627185    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"]
Jul 14 01:36:41.634305 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.634272    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:41.653405 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.653363    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-podinfo\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.653598 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.653416    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-data\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.653598 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.653487    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bp4f\" (UniqueName: \"kubernetes.io/projected/ef367bf9-e985-4d17-a1fb-6687868ca486-kube-api-access-7bp4f\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.653598 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.653542    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-credential-socket\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.653598 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.653562    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-workload-certs\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.653779 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.653612    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-workload-socket\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.653779 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.653648    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-envoy\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.653779 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.653675    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-token\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.653779 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.653728    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef367bf9-e985-4d17-a1fb-6687868ca486-istiod-ca-cert\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.727349 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.727315    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:41.754356 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.754305    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-podinfo\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.754356 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.754366    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-data\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.754652 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.754392    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bp4f\" (UniqueName: \"kubernetes.io/projected/ef367bf9-e985-4d17-a1fb-6687868ca486-kube-api-access-7bp4f\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.754652 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.754422    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-credential-socket\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.754652 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.754437    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-workload-certs\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.754652 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.754468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-workload-socket\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.754860 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.754792    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-envoy\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.754860 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.754853    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-token\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.755017 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.754881    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-data\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.755017 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.754915    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef367bf9-e985-4d17-a1fb-6687868ca486-istiod-ca-cert\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.755017 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.754924    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-credential-socket\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.755017 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.755010    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-workload-certs\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.755390 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.755079    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-workload-socket\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.755580 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.755559    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef367bf9-e985-4d17-a1fb-6687868ca486-istiod-ca-cert\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.757253 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.757232    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-podinfo\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.757380 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.757300    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-envoy\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.761992 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.761941    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bp4f\" (UniqueName: \"kubernetes.io/projected/ef367bf9-e985-4d17-a1fb-6687868ca486-kube-api-access-7bp4f\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.761992 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.761949    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-token\") pod \"e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") " pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:41.929242 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:41.929201    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:42.074473 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:42.074445    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"]
Jul 14 01:36:42.075700 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:36:42.075676    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef367bf9_e985_4d17_a1fb_6687868ca486.slice/crio-71c9bb16b7a1a13e030860a7754fe80c20016211498bfdb7b21e9239fafe42d8 WatchSource:0}: Error finding container 71c9bb16b7a1a13e030860a7754fe80c20016211498bfdb7b21e9239fafe42d8: Status 404 returned error can't find the container with id 71c9bb16b7a1a13e030860a7754fe80c20016211498bfdb7b21e9239fafe42d8
Jul 14 01:36:42.078008 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:42.077978    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:36:42.078079 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:42.078047    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:36:42.078079 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:42.078076    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:36:42.783794 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:42.783752    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc" event={"ID":"ef367bf9-e985-4d17-a1fb-6687868ca486","Type":"ContainerStarted","Data":"76986abd31f4de46494f8c5f93d0e6e98fdf8a5b2979ea76a9e636c47ef9f407"}
Jul 14 01:36:42.784250 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:42.783814    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc" event={"ID":"ef367bf9-e985-4d17-a1fb-6687868ca486","Type":"ContainerStarted","Data":"71c9bb16b7a1a13e030860a7754fe80c20016211498bfdb7b21e9239fafe42d8"}
Jul 14 01:36:42.802488 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:42.802427    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc" podStartSLOduration=1.8024102850000001 podStartE2EDuration="1.802410285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:36:42.800033251 +0000 UTC m=+2232.364442308" watchObservedRunningTime="2026-07-14 01:36:42.802410285 +0000 UTC m=+2232.366819409"
Jul 14 01:36:42.929713 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:42.929673    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:43.935331 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:43.935290    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:44.715138 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.715082    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:44.792230 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.792166    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:44.792877 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.792853    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"]
Jul 14 01:36:44.796440 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.796416    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:44.796603 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.796517    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:36:44.796687 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.796668    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:44.798815 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.798793    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b02b4cdc-serving-cert\""
Jul 14 01:36:44.798977 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.798830    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:36:44.799212 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.799193    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8hlqx\""
Jul 14 01:36:44.799311 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.799245    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:36:44.806701 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.806675    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"]
Jul 14 01:36:44.859185 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.859152    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:44.885304 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.885269    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c70280-7f0e-4c45-9dd0-a650a25b0a63-maas-api-tls\") pod \"maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh\" (UID: \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:36:44.885304 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.885313    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c70280-7f0e-4c45-9dd0-a650a25b0a63-openshift-service-ca\") pod \"maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh\" (UID: \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:36:44.885595 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.885435    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ssx7\" (UniqueName: \"kubernetes.io/projected/52c70280-7f0e-4c45-9dd0-a650a25b0a63-kube-api-access-5ssx7\") pod \"maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh\" (UID: \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:36:44.986682 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.986583    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c70280-7f0e-4c45-9dd0-a650a25b0a63-maas-api-tls\") pod \"maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh\" (UID: \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:36:44.986682 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.986629    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c70280-7f0e-4c45-9dd0-a650a25b0a63-openshift-service-ca\") pod \"maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh\" (UID: \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:36:44.987190 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:36:44.986746    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-serving-cert: secret "maas-api-e2e-mt-b02b4cdc-serving-cert" not found
Jul 14 01:36:44.987190 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.986756    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ssx7\" (UniqueName: \"kubernetes.io/projected/52c70280-7f0e-4c45-9dd0-a650a25b0a63-kube-api-access-5ssx7\") pod \"maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh\" (UID: \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:36:44.987190 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:36:44.986829    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52c70280-7f0e-4c45-9dd0-a650a25b0a63-maas-api-tls podName:52c70280-7f0e-4c45-9dd0-a650a25b0a63 nodeName:}" failed. No retries permitted until 2026-07-14 01:36:45.486807446 +0000 UTC m=+2235.051216485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52c70280-7f0e-4c45-9dd0-a650a25b0a63-maas-api-tls") pod "maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh" (UID: "52c70280-7f0e-4c45-9dd0-a650a25b0a63") : secret "maas-api-e2e-mt-b02b4cdc-serving-cert" not found
Jul 14 01:36:44.987408 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.987389    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c70280-7f0e-4c45-9dd0-a650a25b0a63-openshift-service-ca\") pod \"maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh\" (UID: \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:36:44.995170 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:44.995143    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ssx7\" (UniqueName: \"kubernetes.io/projected/52c70280-7f0e-4c45-9dd0-a650a25b0a63-kube-api-access-5ssx7\") pod \"maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh\" (UID: \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:36:45.267384 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:45.267267    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:45.367520 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:45.367483    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:45.468366 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:45.468334    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:45.491356 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:45.491321    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c70280-7f0e-4c45-9dd0-a650a25b0a63-maas-api-tls\") pod \"maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh\" (UID: \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:36:45.494544 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:45.494514    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c70280-7f0e-4c45-9dd0-a650a25b0a63-maas-api-tls\") pod \"maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh\" (UID: \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:36:45.710014 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:45.709947    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:36:45.856449 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:45.856412    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"]
Jul 14 01:36:45.860231 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:36:45.860200    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c70280_7f0e_4c45_9dd0_a650a25b0a63.slice/crio-363609880b810ee9b23252b653ccdd6aee666518a2fddcf28b87216a5a9ff967 WatchSource:0}: Error finding container 363609880b810ee9b23252b653ccdd6aee666518a2fddcf28b87216a5a9ff967: Status 404 returned error can't find the container with id 363609880b810ee9b23252b653ccdd6aee666518a2fddcf28b87216a5a9ff967
Jul 14 01:36:45.861638 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:45.861622    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 01:36:46.810114 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:46.810066    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh" event={"ID":"52c70280-7f0e-4c45-9dd0-a650a25b0a63","Type":"ContainerStarted","Data":"363609880b810ee9b23252b653ccdd6aee666518a2fddcf28b87216a5a9ff967"}
Jul 14 01:36:47.816791 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:47.816756    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh" event={"ID":"52c70280-7f0e-4c45-9dd0-a650a25b0a63","Type":"ContainerStarted","Data":"2dba195b4bf52b641bc67e7a15651f7dbf45c262f1b6693d01d8370a3e4f9596"}
Jul 14 01:36:47.817215 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:47.816857    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:36:47.841879 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:47.841770    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh" podStartSLOduration=2.112375193 podStartE2EDuration="3.84175083s" totalImagesPullingTime="1.729375637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:36:47.831507401 +0000 UTC m=+2237.395916456" watchObservedRunningTime="2026-07-14 01:36:47.84175083 +0000 UTC m=+2237.406159884"
Jul 14 01:36:49.229881 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:49.229850    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:36:53.827711 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:53.827617    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:36:57.884362 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:57.884325    2567 generic.go:356] "Generic (PLEG): container finished" podID="ef367bf9-e985-4d17-a1fb-6687868ca486" containerID="76986abd31f4de46494f8c5f93d0e6e98fdf8a5b2979ea76a9e636c47ef9f407" exitCode=0
Jul 14 01:36:57.884782 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:57.884402    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc" event={"ID":"ef367bf9-e985-4d17-a1fb-6687868ca486","Type":"ContainerDied","Data":"76986abd31f4de46494f8c5f93d0e6e98fdf8a5b2979ea76a9e636c47ef9f407"}
Jul 14 01:36:57.884824 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:57.884787    2567 scope.go:117] "RemoveContainer" containerID="76986abd31f4de46494f8c5f93d0e6e98fdf8a5b2979ea76a9e636c47ef9f407"
Jul 14 01:36:57.885830 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:57.885802    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:36:57.885903 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:57.885886    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:36:57.885970 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:57.885938    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:36:58.892651 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:58.892613    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc" event={"ID":"ef367bf9-e985-4d17-a1fb-6687868ca486","Type":"ContainerStarted","Data":"28aa40b82798a066d67f5a2b44122a010bafd37d7aead7c5da01cbc68f7a1f0e"}
Jul 14 01:36:58.929682 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:58.929645    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:58.930701 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:58.930670    2567 patch_prober.go:28] interesting pod/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 14 01:36:58.930831 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:58.930731    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc" podUID="ef367bf9-e985-4d17-a1fb-6687868ca486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 14 01:36:59.796341 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:59.796298    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:36:59.930534 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:59.930499    2567 patch_prober.go:28] interesting pod/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 14 01:36:59.930938 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:36:59.930572    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc" podUID="ef367bf9-e985-4d17-a1fb-6687868ca486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 14 01:37:00.930446 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:00.930409    2567 patch_prober.go:28] interesting pod/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 14 01:37:00.930634 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:00.930484    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc" podUID="ef367bf9-e985-4d17-a1fb-6687868ca486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 14 01:37:01.930060 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:01.930019    2567 patch_prober.go:28] interesting pod/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 14 01:37:01.930253 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:01.930091    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc" podUID="ef367bf9-e985-4d17-a1fb-6687868ca486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 14 01:37:02.933812 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:02.933779    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:37:02.934832 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:02.934810    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:37:13.048672 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:13.048633    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:37:48.273071 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:48.273036    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"]
Jul 14 01:37:48.273703 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:48.273325    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh" podUID="52c70280-7f0e-4c45-9dd0-a650a25b0a63" containerName="maas-api" containerID="cri-o://2dba195b4bf52b641bc67e7a15651f7dbf45c262f1b6693d01d8370a3e4f9596" gracePeriod=30
Jul 14 01:37:48.312231 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:48.312186    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:37:48.531491 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:48.531416    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:37:48.660942 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:48.660897    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c70280-7f0e-4c45-9dd0-a650a25b0a63-openshift-service-ca\") pod \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\" (UID: \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\") "
Jul 14 01:37:48.661175 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:48.660995    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c70280-7f0e-4c45-9dd0-a650a25b0a63-maas-api-tls\") pod \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\" (UID: \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\") "
Jul 14 01:37:48.661175 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:48.661053    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ssx7\" (UniqueName: \"kubernetes.io/projected/52c70280-7f0e-4c45-9dd0-a650a25b0a63-kube-api-access-5ssx7\") pod \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\" (UID: \"52c70280-7f0e-4c45-9dd0-a650a25b0a63\") "
Jul 14 01:37:48.661349 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:48.661325    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c70280-7f0e-4c45-9dd0-a650a25b0a63-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52c70280-7f0e-4c45-9dd0-a650a25b0a63" (UID: "52c70280-7f0e-4c45-9dd0-a650a25b0a63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:37:48.663470 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:48.663432    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c70280-7f0e-4c45-9dd0-a650a25b0a63-kube-api-access-5ssx7" (OuterVolumeSpecName: "kube-api-access-5ssx7") pod "52c70280-7f0e-4c45-9dd0-a650a25b0a63" (UID: "52c70280-7f0e-4c45-9dd0-a650a25b0a63"). InnerVolumeSpecName "kube-api-access-5ssx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:37:48.663470 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:48.663441    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c70280-7f0e-4c45-9dd0-a650a25b0a63-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52c70280-7f0e-4c45-9dd0-a650a25b0a63" (UID: "52c70280-7f0e-4c45-9dd0-a650a25b0a63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:37:48.762397 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:48.762357    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c70280-7f0e-4c45-9dd0-a650a25b0a63-maas-api-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:37:48.762397 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:48.762389    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ssx7\" (UniqueName: \"kubernetes.io/projected/52c70280-7f0e-4c45-9dd0-a650a25b0a63-kube-api-access-5ssx7\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:37:48.762397 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:48.762399    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c70280-7f0e-4c45-9dd0-a650a25b0a63-openshift-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:37:49.171446 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:49.171403    2567 generic.go:356] "Generic (PLEG): container finished" podID="52c70280-7f0e-4c45-9dd0-a650a25b0a63" containerID="2dba195b4bf52b641bc67e7a15651f7dbf45c262f1b6693d01d8370a3e4f9596" exitCode=0
Jul 14 01:37:49.171446 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:49.171449    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"
Jul 14 01:37:49.171650 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:49.171487    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh" event={"ID":"52c70280-7f0e-4c45-9dd0-a650a25b0a63","Type":"ContainerDied","Data":"2dba195b4bf52b641bc67e7a15651f7dbf45c262f1b6693d01d8370a3e4f9596"}
Jul 14 01:37:49.171650 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:49.171543    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh" event={"ID":"52c70280-7f0e-4c45-9dd0-a650a25b0a63","Type":"ContainerDied","Data":"363609880b810ee9b23252b653ccdd6aee666518a2fddcf28b87216a5a9ff967"}
Jul 14 01:37:49.171650 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:49.171564    2567 scope.go:117] "RemoveContainer" containerID="2dba195b4bf52b641bc67e7a15651f7dbf45c262f1b6693d01d8370a3e4f9596"
Jul 14 01:37:49.183822 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:49.183793    2567 scope.go:117] "RemoveContainer" containerID="2dba195b4bf52b641bc67e7a15651f7dbf45c262f1b6693d01d8370a3e4f9596"
Jul 14 01:37:49.184201 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:37:49.184181    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dba195b4bf52b641bc67e7a15651f7dbf45c262f1b6693d01d8370a3e4f9596\": container with ID starting with 2dba195b4bf52b641bc67e7a15651f7dbf45c262f1b6693d01d8370a3e4f9596 not found: ID does not exist" containerID="2dba195b4bf52b641bc67e7a15651f7dbf45c262f1b6693d01d8370a3e4f9596"
Jul 14 01:37:49.184288 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:49.184213    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dba195b4bf52b641bc67e7a15651f7dbf45c262f1b6693d01d8370a3e4f9596"} err="failed to get container status \"2dba195b4bf52b641bc67e7a15651f7dbf45c262f1b6693d01d8370a3e4f9596\": rpc error: code = NotFound desc = could not find container \"2dba195b4bf52b641bc67e7a15651f7dbf45c262f1b6693d01d8370a3e4f9596\": container with ID starting with 2dba195b4bf52b641bc67e7a15651f7dbf45c262f1b6693d01d8370a3e4f9596 not found: ID does not exist"
Jul 14 01:37:49.188503 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:49.188470    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"]
Jul 14 01:37:49.191911 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:49.191884    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b02b4cdc-5fdbd699b6-n82qh"]
Jul 14 01:37:50.979158 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:50.977150    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52c70280-7f0e-4c45-9dd0-a650a25b0a63" path="/var/lib/kubelet/pods/52c70280-7f0e-4c45-9dd0-a650a25b0a63/volumes"
Jul 14 01:37:52.923546 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:52.923515    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:37:57.947692 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:37:57.947639    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:38:31.189510 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:38:31.189427    2567 scope.go:117] "RemoveContainer" containerID="4fffe011b6c293ddf121664b55038c0e2dd85a0f4dccbeda22bdb293e57904dd"
Jul 14 01:38:34.279690 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:38:34.279656    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:38:34.451563 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:38:34.451523    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:38:34.503111 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:38:34.503076    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:38:34.925574 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:38:34.925526    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:38:40.337719 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:38:40.337676    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:38:54.302093 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:38:54.302059    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:38:55.510141 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:38:55.510102    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:05.123621 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:05.123585    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"]
Jul 14 01:39:05.124249 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:05.123810    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc" podUID="ef367bf9-e985-4d17-a1fb-6687868ca486" containerName="istio-proxy" containerID="cri-o://28aa40b82798a066d67f5a2b44122a010bafd37d7aead7c5da01cbc68f7a1f0e" gracePeriod=30
Jul 14 01:39:05.144947 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:05.144913    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:05.365329 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:05.365283    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:06.415348 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:06.415307    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:06.512866 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:06.512826    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:06.612928 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:06.612899    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:07.289681 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:07.289643    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:07.383398 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:07.383364    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:07.488287 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:07.488251    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:08.821048 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:08.821016    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:09.595703 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:09.595668    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:09.697666 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:09.697635    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:10.382839 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.382813    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:39:10.477752 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.477661    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-workload-certs\") pod \"ef367bf9-e985-4d17-a1fb-6687868ca486\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") "
Jul 14 01:39:10.477752 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.477697    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-credential-socket\") pod \"ef367bf9-e985-4d17-a1fb-6687868ca486\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") "
Jul 14 01:39:10.477752 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.477723    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-data\") pod \"ef367bf9-e985-4d17-a1fb-6687868ca486\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") "
Jul 14 01:39:10.477752 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.477754    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef367bf9-e985-4d17-a1fb-6687868ca486-istiod-ca-cert\") pod \"ef367bf9-e985-4d17-a1fb-6687868ca486\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") "
Jul 14 01:39:10.478123 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.477776    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bp4f\" (UniqueName: \"kubernetes.io/projected/ef367bf9-e985-4d17-a1fb-6687868ca486-kube-api-access-7bp4f\") pod \"ef367bf9-e985-4d17-a1fb-6687868ca486\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") "
Jul 14 01:39:10.478123 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.477808    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-podinfo\") pod \"ef367bf9-e985-4d17-a1fb-6687868ca486\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") "
Jul 14 01:39:10.478123 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.477833    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-workload-socket\") pod \"ef367bf9-e985-4d17-a1fb-6687868ca486\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") "
Jul 14 01:39:10.478123 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.478028    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-envoy\") pod \"ef367bf9-e985-4d17-a1fb-6687868ca486\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") "
Jul 14 01:39:10.478123 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.478039    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ef367bf9-e985-4d17-a1fb-6687868ca486" (UID: "ef367bf9-e985-4d17-a1fb-6687868ca486"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:39:10.478123 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.478056    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ef367bf9-e985-4d17-a1fb-6687868ca486" (UID: "ef367bf9-e985-4d17-a1fb-6687868ca486"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:39:10.478123 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.478085    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-token\") pod \"ef367bf9-e985-4d17-a1fb-6687868ca486\" (UID: \"ef367bf9-e985-4d17-a1fb-6687868ca486\") "
Jul 14 01:39:10.478500 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.478179    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ef367bf9-e985-4d17-a1fb-6687868ca486" (UID: "ef367bf9-e985-4d17-a1fb-6687868ca486"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:39:10.478500 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.478421    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-workload-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:39:10.478500 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.478444    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-credential-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:39:10.478500 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.478459    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-workload-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:39:10.478778 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.478533    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef367bf9-e985-4d17-a1fb-6687868ca486-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ef367bf9-e985-4d17-a1fb-6687868ca486" (UID: "ef367bf9-e985-4d17-a1fb-6687868ca486"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:39:10.479111 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.479078    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-data" (OuterVolumeSpecName: "istio-data") pod "ef367bf9-e985-4d17-a1fb-6687868ca486" (UID: "ef367bf9-e985-4d17-a1fb-6687868ca486"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:39:10.480431 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.480405    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef367bf9-e985-4d17-a1fb-6687868ca486-kube-api-access-7bp4f" (OuterVolumeSpecName: "kube-api-access-7bp4f") pod "ef367bf9-e985-4d17-a1fb-6687868ca486" (UID: "ef367bf9-e985-4d17-a1fb-6687868ca486"). InnerVolumeSpecName "kube-api-access-7bp4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:39:10.480431 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.480423    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-token" (OuterVolumeSpecName: "istio-token") pod "ef367bf9-e985-4d17-a1fb-6687868ca486" (UID: "ef367bf9-e985-4d17-a1fb-6687868ca486"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:39:10.480799 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.480782    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ef367bf9-e985-4d17-a1fb-6687868ca486" (UID: "ef367bf9-e985-4d17-a1fb-6687868ca486"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:39:10.480863 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.480800    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ef367bf9-e985-4d17-a1fb-6687868ca486" (UID: "ef367bf9-e985-4d17-a1fb-6687868ca486"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 01:39:10.579732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.579677    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-podinfo\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:39:10.579732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.579722    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-envoy\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:39:10.579732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.579732    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-token\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:39:10.579732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.579739    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef367bf9-e985-4d17-a1fb-6687868ca486-istio-data\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:39:10.579732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.579747    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef367bf9-e985-4d17-a1fb-6687868ca486-istiod-ca-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:39:10.580081 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.579755    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bp4f\" (UniqueName: \"kubernetes.io/projected/ef367bf9-e985-4d17-a1fb-6687868ca486-kube-api-access-7bp4f\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:39:10.619324 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.619290    2567 generic.go:356] "Generic (PLEG): container finished" podID="ef367bf9-e985-4d17-a1fb-6687868ca486" containerID="28aa40b82798a066d67f5a2b44122a010bafd37d7aead7c5da01cbc68f7a1f0e" exitCode=0
Jul 14 01:39:10.619499 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.619338    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"
Jul 14 01:39:10.619499 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.619383    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc" event={"ID":"ef367bf9-e985-4d17-a1fb-6687868ca486","Type":"ContainerDied","Data":"28aa40b82798a066d67f5a2b44122a010bafd37d7aead7c5da01cbc68f7a1f0e"}
Jul 14 01:39:10.619499 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.619450    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc" event={"ID":"ef367bf9-e985-4d17-a1fb-6687868ca486","Type":"ContainerDied","Data":"71c9bb16b7a1a13e030860a7754fe80c20016211498bfdb7b21e9239fafe42d8"}
Jul 14 01:39:10.619499 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.619474    2567 scope.go:117] "RemoveContainer" containerID="28aa40b82798a066d67f5a2b44122a010bafd37d7aead7c5da01cbc68f7a1f0e"
Jul 14 01:39:10.629227 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.629208    2567 scope.go:117] "RemoveContainer" containerID="76986abd31f4de46494f8c5f93d0e6e98fdf8a5b2979ea76a9e636c47ef9f407"
Jul 14 01:39:10.638167 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.638151    2567 scope.go:117] "RemoveContainer" containerID="28aa40b82798a066d67f5a2b44122a010bafd37d7aead7c5da01cbc68f7a1f0e"
Jul 14 01:39:10.638451 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:39:10.638433    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28aa40b82798a066d67f5a2b44122a010bafd37d7aead7c5da01cbc68f7a1f0e\": container with ID starting with 28aa40b82798a066d67f5a2b44122a010bafd37d7aead7c5da01cbc68f7a1f0e not found: ID does not exist" containerID="28aa40b82798a066d67f5a2b44122a010bafd37d7aead7c5da01cbc68f7a1f0e"
Jul 14 01:39:10.638491 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.638462    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28aa40b82798a066d67f5a2b44122a010bafd37d7aead7c5da01cbc68f7a1f0e"} err="failed to get container status \"28aa40b82798a066d67f5a2b44122a010bafd37d7aead7c5da01cbc68f7a1f0e\": rpc error: code = NotFound desc = could not find container \"28aa40b82798a066d67f5a2b44122a010bafd37d7aead7c5da01cbc68f7a1f0e\": container with ID starting with 28aa40b82798a066d67f5a2b44122a010bafd37d7aead7c5da01cbc68f7a1f0e not found: ID does not exist"
Jul 14 01:39:10.638491 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.638480    2567 scope.go:117] "RemoveContainer" containerID="76986abd31f4de46494f8c5f93d0e6e98fdf8a5b2979ea76a9e636c47ef9f407"
Jul 14 01:39:10.638725 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:39:10.638696    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76986abd31f4de46494f8c5f93d0e6e98fdf8a5b2979ea76a9e636c47ef9f407\": container with ID starting with 76986abd31f4de46494f8c5f93d0e6e98fdf8a5b2979ea76a9e636c47ef9f407 not found: ID does not exist" containerID="76986abd31f4de46494f8c5f93d0e6e98fdf8a5b2979ea76a9e636c47ef9f407"
Jul 14 01:39:10.638767 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.638723    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76986abd31f4de46494f8c5f93d0e6e98fdf8a5b2979ea76a9e636c47ef9f407"} err="failed to get container status \"76986abd31f4de46494f8c5f93d0e6e98fdf8a5b2979ea76a9e636c47ef9f407\": rpc error: code = NotFound desc = could not find container \"76986abd31f4de46494f8c5f93d0e6e98fdf8a5b2979ea76a9e636c47ef9f407\": container with ID starting with 76986abd31f4de46494f8c5f93d0e6e98fdf8a5b2979ea76a9e636c47ef9f407 not found: ID does not exist"
Jul 14 01:39:10.643371 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.643345    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"]
Jul 14 01:39:10.646835 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.646809    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b02b4cdc-openshift-default-7cc7794fd7-pjsrc"]
Jul 14 01:39:10.976599 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:10.976565    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef367bf9-e985-4d17-a1fb-6687868ca486" path="/var/lib/kubelet/pods/ef367bf9-e985-4d17-a1fb-6687868ca486/volumes"
Jul 14 01:39:14.432618 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:14.432587    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:14.491749 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:14.491713    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:15.092395 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:15.092355    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:20.896703 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:20.896668    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:21.001601 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:21.001567    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:21.096570 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:21.096532    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:31.214879 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:31.214843    2567 scope.go:117] "RemoveContainer" containerID="e79675677ef34b2c571a36ff72eb3c12fa653e8840901c7eea0cb652e547e2a9"
Jul 14 01:39:31.586168 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:31.586088    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:31.920732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:31.920702    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:36.915380 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:36.915347    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:52.874258 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:52.874166    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:58.446412 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:58.446377    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:58.508898 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:58.508860    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:58.783335 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:58.783244    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:59.066675 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.066588    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"]
Jul 14 01:39:59.066999 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.066984    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef367bf9-e985-4d17-a1fb-6687868ca486" containerName="istio-proxy"
Jul 14 01:39:59.067067 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.067001    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef367bf9-e985-4d17-a1fb-6687868ca486" containerName="istio-proxy"
Jul 14 01:39:59.067067 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.067013    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef367bf9-e985-4d17-a1fb-6687868ca486" containerName="istio-proxy"
Jul 14 01:39:59.067067 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.067019    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef367bf9-e985-4d17-a1fb-6687868ca486" containerName="istio-proxy"
Jul 14 01:39:59.067067 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.067030    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c70280-7f0e-4c45-9dd0-a650a25b0a63" containerName="maas-api"
Jul 14 01:39:59.067067 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.067036    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c70280-7f0e-4c45-9dd0-a650a25b0a63" containerName="maas-api"
Jul 14 01:39:59.067228 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.067096    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef367bf9-e985-4d17-a1fb-6687868ca486" containerName="istio-proxy"
Jul 14 01:39:59.067228 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.067105    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c70280-7f0e-4c45-9dd0-a650a25b0a63" containerName="maas-api"
Jul 14 01:39:59.070075 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.070056    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.072034 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.072016    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8ddee0-openshift-default-dockercfg-xxslk\""
Jul 14 01:39:59.080345 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.080315    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"]
Jul 14 01:39:59.084751 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.084713    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:59.116271 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.116231    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-credential-socket\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.116271 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.116273    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istiod-ca-cert\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.116506 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.116309    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-token\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.116506 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.116371    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-workload-certs\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.116506 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.116402    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-podinfo\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.116506 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.116467    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-data\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.116651 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.116506    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8c8t\" (UniqueName: \"kubernetes.io/projected/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-kube-api-access-g8c8t\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.116651 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.116534    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-envoy\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.116651 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.116552    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-workload-socket\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.217261 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.217219    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8c8t\" (UniqueName: \"kubernetes.io/projected/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-kube-api-access-g8c8t\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.217261 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.217267    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-envoy\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.217538 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.217286    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-workload-socket\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.217538 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.217311    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-credential-socket\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.217538 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.217357    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istiod-ca-cert\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.217538 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.217401    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-token\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.217754 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.217435    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-workload-certs\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.217818 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.217774    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-podinfo\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.217818 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.217800    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-workload-socket\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.217927 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.217824    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-credential-socket\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.218016 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.217902    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-data\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.218016 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.217923    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-workload-certs\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.218294 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.218267    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-data\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.218294 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.218290    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istiod-ca-cert\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.220281 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.220263    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-envoy\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.220544 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.220524    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-podinfo\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.224811 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.224780    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-token\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.225405 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.225367    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8c8t\" (UniqueName: \"kubernetes.io/projected/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-kube-api-access-g8c8t\") pod \"e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") " pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.309356 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.309323    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:39:59.381033 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.380997    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:39:59.523299 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.523270    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"]
Jul 14 01:39:59.524642 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:39:59.524602    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b53a4ed_6863_4bf4_bebd_3f265fdf2dce.slice/crio-1244995fa0b214437c9fc0f09b3f5e9bf0e6e1c2103663c30b5ad51d3309bdec WatchSource:0}: Error finding container 1244995fa0b214437c9fc0f09b3f5e9bf0e6e1c2103663c30b5ad51d3309bdec: Status 404 returned error can't find the container with id 1244995fa0b214437c9fc0f09b3f5e9bf0e6e1c2103663c30b5ad51d3309bdec
Jul 14 01:39:59.526600 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.526564    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:39:59.526680 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.526627    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:39:59.526680 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.526657    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:39:59.877932 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.877890    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm" event={"ID":"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce","Type":"ContainerStarted","Data":"224b232aec3e65b8b60cf1b45f16ebe87d74ee17e3f86b8dd1a8f8611a12c56c"}
Jul 14 01:39:59.878142 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.877949    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm" event={"ID":"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce","Type":"ContainerStarted","Data":"1244995fa0b214437c9fc0f09b3f5e9bf0e6e1c2103663c30b5ad51d3309bdec"}
Jul 14 01:39:59.896799 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:39:59.896745    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm" podStartSLOduration=0.896728238 podStartE2EDuration="896.728238ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:39:59.895243684 +0000 UTC m=+2429.459652752" watchObservedRunningTime="2026-07-14 01:39:59.896728238 +0000 UTC m=+2429.461137292"
Jul 14 01:40:00.382026 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:00.381989    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:40:00.386685 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:00.386656    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:40:00.883097 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:00.883067    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:40:00.884011 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:00.883990    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:40:01.813758 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:01.813721    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:01.890867 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:01.890833    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:01.897609 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:01.897579    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"]
Jul 14 01:40:01.898257 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:01.898237    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef367bf9-e985-4d17-a1fb-6687868ca486" containerName="istio-proxy"
Jul 14 01:40:01.907988 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:01.907941    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:40:01.910183 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:01.910137    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"]
Jul 14 01:40:01.910684 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:01.910495    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8ddee0-serving-cert\""
Jul 14 01:40:01.910684 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:01.910522    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:40:01.910684 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:01.910586    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8hlqx\""
Jul 14 01:40:01.911220 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:01.911201    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:40:01.941813 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:01.941778    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78ae992-8b27-4368-b38a-51c6e26a903a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt\" (UID: \"f78ae992-8b27-4368-b38a-51c6e26a903a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:40:01.942041 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:01.941833    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs6r2\" (UniqueName: \"kubernetes.io/projected/f78ae992-8b27-4368-b38a-51c6e26a903a-kube-api-access-rs6r2\") pod \"maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt\" (UID: \"f78ae992-8b27-4368-b38a-51c6e26a903a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:40:01.942041 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:01.942031    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f78ae992-8b27-4368-b38a-51c6e26a903a-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt\" (UID: \"f78ae992-8b27-4368-b38a-51c6e26a903a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:40:01.966237 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:01.966206    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:02.043062 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:02.043015    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78ae992-8b27-4368-b38a-51c6e26a903a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt\" (UID: \"f78ae992-8b27-4368-b38a-51c6e26a903a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:40:02.043062 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:02.043066    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs6r2\" (UniqueName: \"kubernetes.io/projected/f78ae992-8b27-4368-b38a-51c6e26a903a-kube-api-access-rs6r2\") pod \"maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt\" (UID: \"f78ae992-8b27-4368-b38a-51c6e26a903a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:40:02.043354 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:02.043131    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f78ae992-8b27-4368-b38a-51c6e26a903a-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt\" (UID: \"f78ae992-8b27-4368-b38a-51c6e26a903a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:40:02.043354 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:40:02.043237    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-serving-cert: secret "maas-api-e2e-inf-a-8ddee0-serving-cert" not found
Jul 14 01:40:02.043354 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:40:02.043325    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f78ae992-8b27-4368-b38a-51c6e26a903a-maas-api-tls podName:f78ae992-8b27-4368-b38a-51c6e26a903a nodeName:}" failed. No retries permitted until 2026-07-14 01:40:02.543308694 +0000 UTC m=+2432.107717726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f78ae992-8b27-4368-b38a-51c6e26a903a-maas-api-tls") pod "maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt" (UID: "f78ae992-8b27-4368-b38a-51c6e26a903a") : secret "maas-api-e2e-inf-a-8ddee0-serving-cert" not found
Jul 14 01:40:02.043762 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:02.043736    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78ae992-8b27-4368-b38a-51c6e26a903a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt\" (UID: \"f78ae992-8b27-4368-b38a-51c6e26a903a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:40:02.051858 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:02.051817    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs6r2\" (UniqueName: \"kubernetes.io/projected/f78ae992-8b27-4368-b38a-51c6e26a903a-kube-api-access-rs6r2\") pod \"maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt\" (UID: \"f78ae992-8b27-4368-b38a-51c6e26a903a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:40:02.269514 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:02.269477    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:02.547384 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:02.547278    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f78ae992-8b27-4368-b38a-51c6e26a903a-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt\" (UID: \"f78ae992-8b27-4368-b38a-51c6e26a903a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:40:02.550013 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:02.549988    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f78ae992-8b27-4368-b38a-51c6e26a903a-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt\" (UID: \"f78ae992-8b27-4368-b38a-51c6e26a903a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:40:02.556096 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:02.556064    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:02.657017 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:02.656979    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:02.820384 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:02.820284    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:40:02.965011 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:02.964983    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"]
Jul 14 01:40:02.966899 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:40:02.966875    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78ae992_8b27_4368_b38a_51c6e26a903a.slice/crio-1d6c612dbf57b0ae97419f1d75f8d35100ebe2bd5c690426028b04576eebd5c0 WatchSource:0}: Error finding container 1d6c612dbf57b0ae97419f1d75f8d35100ebe2bd5c690426028b04576eebd5c0: Status 404 returned error can't find the container with id 1d6c612dbf57b0ae97419f1d75f8d35100ebe2bd5c690426028b04576eebd5c0
Jul 14 01:40:03.902444 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:03.902405    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt" event={"ID":"f78ae992-8b27-4368-b38a-51c6e26a903a","Type":"ContainerStarted","Data":"37608b4ca69ba35c20904f87b509ef07469805a50af492fdeae34cb6170054e2"}
Jul 14 01:40:03.902628 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:03.902499    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt" event={"ID":"f78ae992-8b27-4368-b38a-51c6e26a903a","Type":"ContainerStarted","Data":"1d6c612dbf57b0ae97419f1d75f8d35100ebe2bd5c690426028b04576eebd5c0"}
Jul 14 01:40:03.902628 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:03.902555    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:40:03.920803 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:03.920749    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt" podStartSLOduration=2.36281654 podStartE2EDuration="2.920734599s" totalImagesPullingTime="557.918059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:40:03.918195874 +0000 UTC m=+2433.482604929" watchObservedRunningTime="2026-07-14 01:40:03.920734599 +0000 UTC m=+2433.485143653"
Jul 14 01:40:06.307947 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:06.307918    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:06.631221 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:06.631185    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:06.922927 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:06.922812    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:07.203154 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:07.203069    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:09.914525 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:09.914495    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:40:10.422315 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.422282    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:10.522131 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.522065    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:10.650228 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.650191    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"]
Jul 14 01:40:10.654014 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.653988    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:40:10.655716 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.655690    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0c7042-serving-cert\""
Jul 14 01:40:10.665390 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.665337    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"]
Jul 14 01:40:10.689739 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.689658    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:10.720824 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.720781    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2818938-bd95-4366-b9a3-7ac35bb1f749-maas-api-tls\") pod \"maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2\" (UID: \"d2818938-bd95-4366-b9a3-7ac35bb1f749\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:40:10.721039 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.720868    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbsfv\" (UniqueName: \"kubernetes.io/projected/d2818938-bd95-4366-b9a3-7ac35bb1f749-kube-api-access-dbsfv\") pod \"maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2\" (UID: \"d2818938-bd95-4366-b9a3-7ac35bb1f749\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:40:10.721039 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.720915    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2818938-bd95-4366-b9a3-7ac35bb1f749-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2\" (UID: \"d2818938-bd95-4366-b9a3-7ac35bb1f749\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:40:10.821823 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.821771    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2818938-bd95-4366-b9a3-7ac35bb1f749-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2\" (UID: \"d2818938-bd95-4366-b9a3-7ac35bb1f749\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:40:10.822078 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.821844    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2818938-bd95-4366-b9a3-7ac35bb1f749-maas-api-tls\") pod \"maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2\" (UID: \"d2818938-bd95-4366-b9a3-7ac35bb1f749\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:40:10.822078 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.821915    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbsfv\" (UniqueName: \"kubernetes.io/projected/d2818938-bd95-4366-b9a3-7ac35bb1f749-kube-api-access-dbsfv\") pod \"maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2\" (UID: \"d2818938-bd95-4366-b9a3-7ac35bb1f749\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:40:10.822078 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:40:10.821999    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-serving-cert: secret "maas-api-e2e-inf-b-0c7042-serving-cert" not found
Jul 14 01:40:10.822220 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:40:10.822096    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2818938-bd95-4366-b9a3-7ac35bb1f749-maas-api-tls podName:d2818938-bd95-4366-b9a3-7ac35bb1f749 nodeName:}" failed. No retries permitted until 2026-07-14 01:40:11.322074085 +0000 UTC m=+2440.886483131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2818938-bd95-4366-b9a3-7ac35bb1f749-maas-api-tls") pod "maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2" (UID: "d2818938-bd95-4366-b9a3-7ac35bb1f749") : secret "maas-api-e2e-inf-b-0c7042-serving-cert" not found
Jul 14 01:40:10.822593 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.822575    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2818938-bd95-4366-b9a3-7ac35bb1f749-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2\" (UID: \"d2818938-bd95-4366-b9a3-7ac35bb1f749\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:40:10.832041 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.832002    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbsfv\" (UniqueName: \"kubernetes.io/projected/d2818938-bd95-4366-b9a3-7ac35bb1f749-kube-api-access-dbsfv\") pod \"maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2\" (UID: \"d2818938-bd95-4366-b9a3-7ac35bb1f749\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:40:10.906204 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:10.906157    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:11.219613 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:11.219579    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:11.321899 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:11.321864    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:11.325746 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:11.325721    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2818938-bd95-4366-b9a3-7ac35bb1f749-maas-api-tls\") pod \"maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2\" (UID: \"d2818938-bd95-4366-b9a3-7ac35bb1f749\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:40:11.328513 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:11.328488    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2818938-bd95-4366-b9a3-7ac35bb1f749-maas-api-tls\") pod \"maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2\" (UID: \"d2818938-bd95-4366-b9a3-7ac35bb1f749\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:40:11.565848 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:11.565752    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:40:11.732243 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:11.732205    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"]
Jul 14 01:40:11.961208 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:11.961162    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2" event={"ID":"d2818938-bd95-4366-b9a3-7ac35bb1f749","Type":"ContainerStarted","Data":"dfccf10bef5dcb76bd710508a1e0a60b8cfb57859307b5def412e3a7daa2aee8"}
Jul 14 01:40:12.967818 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:12.967786    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2" event={"ID":"d2818938-bd95-4366-b9a3-7ac35bb1f749","Type":"ContainerStarted","Data":"d4df2230c731706f5fda915f8a9788a478801ce12f347f276414eb1c923293ed"}
Jul 14 01:40:12.968266 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:12.967979    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:40:12.982676 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:12.982619    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2" podStartSLOduration=2.661386798 podStartE2EDuration="2.982597523s" totalImagesPullingTime="321.210725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:40:12.98129428 +0000 UTC m=+2442.545703335" watchObservedRunningTime="2026-07-14 01:40:12.982597523 +0000 UTC m=+2442.547006604"
Jul 14 01:40:14.826788 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:14.826749    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:16.925143 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:16.925101    2567 patch_prober.go:28] interesting pod/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 14 01:40:16.925562 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:16.925167    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm" podUID="0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 14 01:40:17.994715 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:17.994673    2567 generic.go:356] "Generic (PLEG): container finished" podID="0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" containerID="224b232aec3e65b8b60cf1b45f16ebe87d74ee17e3f86b8dd1a8f8611a12c56c" exitCode=0
Jul 14 01:40:17.995180 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:17.994750    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm" event={"ID":"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce","Type":"ContainerDied","Data":"224b232aec3e65b8b60cf1b45f16ebe87d74ee17e3f86b8dd1a8f8611a12c56c"}
Jul 14 01:40:17.995180 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:17.995175    2567 scope.go:117] "RemoveContainer" containerID="224b232aec3e65b8b60cf1b45f16ebe87d74ee17e3f86b8dd1a8f8611a12c56c"
Jul 14 01:40:17.996259 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:17.996232    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:40:17.996318 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:17.996302    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:40:17.996358 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:17.996347    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:40:18.979509 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:18.979483    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:40:19.003429 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:19.003395    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm" event={"ID":"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce","Type":"ContainerStarted","Data":"03ba4b0b06585b2e56786302c9b3e5705ac22d08f9641639a511a6e7173b5dbc"}
Jul 14 01:40:19.381310 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:19.381274    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:40:19.381572 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:19.381548    2567 patch_prober.go:28] interesting pod/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Jul 14 01:40:19.381667 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:19.381619    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm" podUID="0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Jul 14 01:40:20.381784 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:20.381744    2567 patch_prober.go:28] interesting pod/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Jul 14 01:40:20.382220 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:20.381804    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm" podUID="0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Jul 14 01:40:21.382528 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:21.382487    2567 patch_prober.go:28] interesting pod/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Jul 14 01:40:21.382944 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:21.382560    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm" podUID="0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Jul 14 01:40:22.381760 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:22.381718    2567 patch_prober.go:28] interesting pod/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Jul 14 01:40:22.381940 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:22.381776    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm" podUID="0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Jul 14 01:40:23.385169 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:23.385136    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:40:23.385538 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:23.385415    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:40:23.386230 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:23.386213    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:40:23.447671 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:23.447622    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:23.548142 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:23.548111    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:30.673486 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:30.673444    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:31.938263 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:31.938229    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:32.040971 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:32.040924    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:40:58.180158 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:40:58.180113    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:41:25.835574 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:25.835463    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:41:25.936799 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:25.936768    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:41:28.548883 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:28.548848    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:41:29.505148 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:29.505102    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:41:30.317351 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:30.317320    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:41:30.417379 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:30.417347    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:41:40.009478 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.009445    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"]
Jul 14 01:41:40.009909 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.009697    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt" podUID="f78ae992-8b27-4368-b38a-51c6e26a903a" containerName="maas-api" containerID="cri-o://37608b4ca69ba35c20904f87b509ef07469805a50af492fdeae34cb6170054e2" gracePeriod=30
Jul 14 01:41:40.029584 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.029549    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:41:40.273800 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.273735    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:41:40.327860 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.327821    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs6r2\" (UniqueName: \"kubernetes.io/projected/f78ae992-8b27-4368-b38a-51c6e26a903a-kube-api-access-rs6r2\") pod \"f78ae992-8b27-4368-b38a-51c6e26a903a\" (UID: \"f78ae992-8b27-4368-b38a-51c6e26a903a\") "
Jul 14 01:41:40.327860 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.327862    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f78ae992-8b27-4368-b38a-51c6e26a903a-maas-api-tls\") pod \"f78ae992-8b27-4368-b38a-51c6e26a903a\" (UID: \"f78ae992-8b27-4368-b38a-51c6e26a903a\") "
Jul 14 01:41:40.328105 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.327974    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78ae992-8b27-4368-b38a-51c6e26a903a-openshift-service-ca\") pod \"f78ae992-8b27-4368-b38a-51c6e26a903a\" (UID: \"f78ae992-8b27-4368-b38a-51c6e26a903a\") "
Jul 14 01:41:40.328358 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.328331    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78ae992-8b27-4368-b38a-51c6e26a903a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f78ae992-8b27-4368-b38a-51c6e26a903a" (UID: "f78ae992-8b27-4368-b38a-51c6e26a903a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:41:40.330138 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.330113    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f78ae992-8b27-4368-b38a-51c6e26a903a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f78ae992-8b27-4368-b38a-51c6e26a903a" (UID: "f78ae992-8b27-4368-b38a-51c6e26a903a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:41:40.330234 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.330147    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78ae992-8b27-4368-b38a-51c6e26a903a-kube-api-access-rs6r2" (OuterVolumeSpecName: "kube-api-access-rs6r2") pod "f78ae992-8b27-4368-b38a-51c6e26a903a" (UID: "f78ae992-8b27-4368-b38a-51c6e26a903a"). InnerVolumeSpecName "kube-api-access-rs6r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:41:40.429191 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.429144    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs6r2\" (UniqueName: \"kubernetes.io/projected/f78ae992-8b27-4368-b38a-51c6e26a903a-kube-api-access-rs6r2\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:41:40.429191 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.429180    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f78ae992-8b27-4368-b38a-51c6e26a903a-maas-api-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:41:40.429191 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.429190    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78ae992-8b27-4368-b38a-51c6e26a903a-openshift-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:41:40.448356 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.448316    2567 generic.go:356] "Generic (PLEG): container finished" podID="f78ae992-8b27-4368-b38a-51c6e26a903a" containerID="37608b4ca69ba35c20904f87b509ef07469805a50af492fdeae34cb6170054e2" exitCode=0
Jul 14 01:41:40.448356 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.448359    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"
Jul 14 01:41:40.448591 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.448369    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt" event={"ID":"f78ae992-8b27-4368-b38a-51c6e26a903a","Type":"ContainerDied","Data":"37608b4ca69ba35c20904f87b509ef07469805a50af492fdeae34cb6170054e2"}
Jul 14 01:41:40.448591 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.448423    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt" event={"ID":"f78ae992-8b27-4368-b38a-51c6e26a903a","Type":"ContainerDied","Data":"1d6c612dbf57b0ae97419f1d75f8d35100ebe2bd5c690426028b04576eebd5c0"}
Jul 14 01:41:40.448591 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.448449    2567 scope.go:117] "RemoveContainer" containerID="37608b4ca69ba35c20904f87b509ef07469805a50af492fdeae34cb6170054e2"
Jul 14 01:41:40.457857 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.457836    2567 scope.go:117] "RemoveContainer" containerID="37608b4ca69ba35c20904f87b509ef07469805a50af492fdeae34cb6170054e2"
Jul 14 01:41:40.458192 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:41:40.458174    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37608b4ca69ba35c20904f87b509ef07469805a50af492fdeae34cb6170054e2\": container with ID starting with 37608b4ca69ba35c20904f87b509ef07469805a50af492fdeae34cb6170054e2 not found: ID does not exist" containerID="37608b4ca69ba35c20904f87b509ef07469805a50af492fdeae34cb6170054e2"
Jul 14 01:41:40.458265 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.458206    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37608b4ca69ba35c20904f87b509ef07469805a50af492fdeae34cb6170054e2"} err="failed to get container status \"37608b4ca69ba35c20904f87b509ef07469805a50af492fdeae34cb6170054e2\": rpc error: code = NotFound desc = could not find container \"37608b4ca69ba35c20904f87b509ef07469805a50af492fdeae34cb6170054e2\": container with ID starting with 37608b4ca69ba35c20904f87b509ef07469805a50af492fdeae34cb6170054e2 not found: ID does not exist"
Jul 14 01:41:40.467662 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.467636    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"]
Jul 14 01:41:40.471730 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.471704    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ddee0-5f547c9f7d-bjkrt"]
Jul 14 01:41:40.975036 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:40.974996    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f78ae992-8b27-4368-b38a-51c6e26a903a" path="/var/lib/kubelet/pods/f78ae992-8b27-4368-b38a-51c6e26a903a/volumes"
Jul 14 01:41:44.711581 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:44.711548    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:41:50.345366 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:50.345327    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:41:50.403915 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:50.403879    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:41:55.377075 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:55.377023    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"]
Jul 14 01:41:55.395684 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:55.395640    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:41:55.536224 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:55.536179    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm" podUID="0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" containerName="istio-proxy" containerID="cri-o://03ba4b0b06585b2e56786302c9b3e5705ac22d08f9641639a511a6e7173b5dbc" gracePeriod=30
Jul 14 01:41:55.627574 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:55.627489    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:41:56.558895 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:56.558861    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:41:56.661617 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:41:56.661579    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:00.788434 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.788407    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:42:00.910833 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.910790    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-envoy\") pod \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") "
Jul 14 01:42:00.911079 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.910851    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-credential-socket\") pod \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") "
Jul 14 01:42:00.911079 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.910876    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-token\") pod \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") "
Jul 14 01:42:00.911079 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.910934    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-workload-certs\") pod \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") "
Jul 14 01:42:00.911079 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.911004    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istiod-ca-cert\") pod \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") "
Jul 14 01:42:00.911079 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.911035    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-workload-socket\") pod \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") "
Jul 14 01:42:00.911343 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.911080    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-data\") pod \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") "
Jul 14 01:42:00.911343 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.911105    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8c8t\" (UniqueName: \"kubernetes.io/projected/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-kube-api-access-g8c8t\") pod \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") "
Jul 14 01:42:00.911343 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.911159    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-podinfo\") pod \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\" (UID: \"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce\") "
Jul 14 01:42:00.911343 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.911308    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" (UID: "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:42:00.911343 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.911297    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" (UID: "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:42:00.911604 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.911407    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" (UID: "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:42:00.911604 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.911472    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-workload-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:42:00.911604 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.911491    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-workload-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:42:00.911880 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.911832    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" (UID: "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:42:00.912182 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.912154    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" (UID: "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:42:00.913296 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.913265    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" (UID: "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:42:00.913404 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.913295    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" (UID: "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:42:00.913858 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.913838    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-kube-api-access-g8c8t" (OuterVolumeSpecName: "kube-api-access-g8c8t") pod "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" (UID: "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce"). InnerVolumeSpecName "kube-api-access-g8c8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:42:00.913858 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:00.913850    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" (UID: "0b53a4ed-6863-4bf4-bebd-3f265fdf2dce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 01:42:01.012976 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.012921    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-data\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:42:01.012976 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.012971    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8c8t\" (UniqueName: \"kubernetes.io/projected/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-kube-api-access-g8c8t\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:42:01.012976 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.012985    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-podinfo\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:42:01.013222 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.012993    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-envoy\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:42:01.013222 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.013001    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-credential-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:42:01.013222 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.013009    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istio-token\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:42:01.013222 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.013017    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce-istiod-ca-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:42:01.569205 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.569168    2567 generic.go:356] "Generic (PLEG): container finished" podID="0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" containerID="03ba4b0b06585b2e56786302c9b3e5705ac22d08f9641639a511a6e7173b5dbc" exitCode=0
Jul 14 01:42:01.569388 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.569225    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"
Jul 14 01:42:01.569388 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.569269    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm" event={"ID":"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce","Type":"ContainerDied","Data":"03ba4b0b06585b2e56786302c9b3e5705ac22d08f9641639a511a6e7173b5dbc"}
Jul 14 01:42:01.569388 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.569349    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm" event={"ID":"0b53a4ed-6863-4bf4-bebd-3f265fdf2dce","Type":"ContainerDied","Data":"1244995fa0b214437c9fc0f09b3f5e9bf0e6e1c2103663c30b5ad51d3309bdec"}
Jul 14 01:42:01.569388 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.569373    2567 scope.go:117] "RemoveContainer" containerID="03ba4b0b06585b2e56786302c9b3e5705ac22d08f9641639a511a6e7173b5dbc"
Jul 14 01:42:01.578456 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.578436    2567 scope.go:117] "RemoveContainer" containerID="224b232aec3e65b8b60cf1b45f16ebe87d74ee17e3f86b8dd1a8f8611a12c56c"
Jul 14 01:42:01.586017 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.585985    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"]
Jul 14 01:42:01.588070 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.588046    2567 scope.go:117] "RemoveContainer" containerID="03ba4b0b06585b2e56786302c9b3e5705ac22d08f9641639a511a6e7173b5dbc"
Jul 14 01:42:01.588226 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.588209    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8ddee0-openshift-default-6c4c6dd8f7-b9xpm"]
Jul 14 01:42:01.588393 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:42:01.588372    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03ba4b0b06585b2e56786302c9b3e5705ac22d08f9641639a511a6e7173b5dbc\": container with ID starting with 03ba4b0b06585b2e56786302c9b3e5705ac22d08f9641639a511a6e7173b5dbc not found: ID does not exist" containerID="03ba4b0b06585b2e56786302c9b3e5705ac22d08f9641639a511a6e7173b5dbc"
Jul 14 01:42:01.588428 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.588405    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03ba4b0b06585b2e56786302c9b3e5705ac22d08f9641639a511a6e7173b5dbc"} err="failed to get container status \"03ba4b0b06585b2e56786302c9b3e5705ac22d08f9641639a511a6e7173b5dbc\": rpc error: code = NotFound desc = could not find container \"03ba4b0b06585b2e56786302c9b3e5705ac22d08f9641639a511a6e7173b5dbc\": container with ID starting with 03ba4b0b06585b2e56786302c9b3e5705ac22d08f9641639a511a6e7173b5dbc not found: ID does not exist"
Jul 14 01:42:01.588428 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.588424    2567 scope.go:117] "RemoveContainer" containerID="224b232aec3e65b8b60cf1b45f16ebe87d74ee17e3f86b8dd1a8f8611a12c56c"
Jul 14 01:42:01.588646 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:42:01.588630    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"224b232aec3e65b8b60cf1b45f16ebe87d74ee17e3f86b8dd1a8f8611a12c56c\": container with ID starting with 224b232aec3e65b8b60cf1b45f16ebe87d74ee17e3f86b8dd1a8f8611a12c56c not found: ID does not exist" containerID="224b232aec3e65b8b60cf1b45f16ebe87d74ee17e3f86b8dd1a8f8611a12c56c"
Jul 14 01:42:01.588691 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:01.588651    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224b232aec3e65b8b60cf1b45f16ebe87d74ee17e3f86b8dd1a8f8611a12c56c"} err="failed to get container status \"224b232aec3e65b8b60cf1b45f16ebe87d74ee17e3f86b8dd1a8f8611a12c56c\": rpc error: code = NotFound desc = could not find container \"224b232aec3e65b8b60cf1b45f16ebe87d74ee17e3f86b8dd1a8f8611a12c56c\": container with ID starting with 224b232aec3e65b8b60cf1b45f16ebe87d74ee17e3f86b8dd1a8f8611a12c56c not found: ID does not exist"
Jul 14 01:42:02.975016 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:02.974983    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" path="/var/lib/kubelet/pods/0b53a4ed-6863-4bf4-bebd-3f265fdf2dce/volumes"
Jul 14 01:42:06.121117 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:06.121076    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"]
Jul 14 01:42:06.121553 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:06.121394    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2" podUID="d2818938-bd95-4366-b9a3-7ac35bb1f749" containerName="maas-api" containerID="cri-o://d4df2230c731706f5fda915f8a9788a478801ce12f347f276414eb1c923293ed" gracePeriod=30
Jul 14 01:42:06.142183 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:06.142150    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:07.368241 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.368215    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:42:07.470491 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.470390    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbsfv\" (UniqueName: \"kubernetes.io/projected/d2818938-bd95-4366-b9a3-7ac35bb1f749-kube-api-access-dbsfv\") pod \"d2818938-bd95-4366-b9a3-7ac35bb1f749\" (UID: \"d2818938-bd95-4366-b9a3-7ac35bb1f749\") "
Jul 14 01:42:07.470491 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.470437    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2818938-bd95-4366-b9a3-7ac35bb1f749-maas-api-tls\") pod \"d2818938-bd95-4366-b9a3-7ac35bb1f749\" (UID: \"d2818938-bd95-4366-b9a3-7ac35bb1f749\") "
Jul 14 01:42:07.470491 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.470480    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2818938-bd95-4366-b9a3-7ac35bb1f749-openshift-service-ca\") pod \"d2818938-bd95-4366-b9a3-7ac35bb1f749\" (UID: \"d2818938-bd95-4366-b9a3-7ac35bb1f749\") "
Jul 14 01:42:07.470924 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.470894    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2818938-bd95-4366-b9a3-7ac35bb1f749-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d2818938-bd95-4366-b9a3-7ac35bb1f749" (UID: "d2818938-bd95-4366-b9a3-7ac35bb1f749"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:42:07.472600 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.472578    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2818938-bd95-4366-b9a3-7ac35bb1f749-kube-api-access-dbsfv" (OuterVolumeSpecName: "kube-api-access-dbsfv") pod "d2818938-bd95-4366-b9a3-7ac35bb1f749" (UID: "d2818938-bd95-4366-b9a3-7ac35bb1f749"). InnerVolumeSpecName "kube-api-access-dbsfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:42:07.472716 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.472697    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2818938-bd95-4366-b9a3-7ac35bb1f749-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d2818938-bd95-4366-b9a3-7ac35bb1f749" (UID: "d2818938-bd95-4366-b9a3-7ac35bb1f749"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:42:07.572049 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.572012    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbsfv\" (UniqueName: \"kubernetes.io/projected/d2818938-bd95-4366-b9a3-7ac35bb1f749-kube-api-access-dbsfv\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:42:07.572049 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.572043    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2818938-bd95-4366-b9a3-7ac35bb1f749-maas-api-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:42:07.572049 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.572052    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2818938-bd95-4366-b9a3-7ac35bb1f749-openshift-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:42:07.606441 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.606407    2567 generic.go:356] "Generic (PLEG): container finished" podID="d2818938-bd95-4366-b9a3-7ac35bb1f749" containerID="d4df2230c731706f5fda915f8a9788a478801ce12f347f276414eb1c923293ed" exitCode=0
Jul 14 01:42:07.606603 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.606467    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"
Jul 14 01:42:07.606603 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.606499    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2" event={"ID":"d2818938-bd95-4366-b9a3-7ac35bb1f749","Type":"ContainerDied","Data":"d4df2230c731706f5fda915f8a9788a478801ce12f347f276414eb1c923293ed"}
Jul 14 01:42:07.606603 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.606566    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2" event={"ID":"d2818938-bd95-4366-b9a3-7ac35bb1f749","Type":"ContainerDied","Data":"dfccf10bef5dcb76bd710508a1e0a60b8cfb57859307b5def412e3a7daa2aee8"}
Jul 14 01:42:07.606603 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.606598    2567 scope.go:117] "RemoveContainer" containerID="d4df2230c731706f5fda915f8a9788a478801ce12f347f276414eb1c923293ed"
Jul 14 01:42:07.616508 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.616489    2567 scope.go:117] "RemoveContainer" containerID="d4df2230c731706f5fda915f8a9788a478801ce12f347f276414eb1c923293ed"
Jul 14 01:42:07.616766 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:42:07.616744    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4df2230c731706f5fda915f8a9788a478801ce12f347f276414eb1c923293ed\": container with ID starting with d4df2230c731706f5fda915f8a9788a478801ce12f347f276414eb1c923293ed not found: ID does not exist" containerID="d4df2230c731706f5fda915f8a9788a478801ce12f347f276414eb1c923293ed"
Jul 14 01:42:07.616819 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.616780    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4df2230c731706f5fda915f8a9788a478801ce12f347f276414eb1c923293ed"} err="failed to get container status \"d4df2230c731706f5fda915f8a9788a478801ce12f347f276414eb1c923293ed\": rpc error: code = NotFound desc = could not find container \"d4df2230c731706f5fda915f8a9788a478801ce12f347f276414eb1c923293ed\": container with ID starting with d4df2230c731706f5fda915f8a9788a478801ce12f347f276414eb1c923293ed not found: ID does not exist"
Jul 14 01:42:07.627611 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.627586    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"]
Jul 14 01:42:07.631474 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:07.631447    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0c7042-657c759c8c-99dp2"]
Jul 14 01:42:08.974665 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:08.974634    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2818938-bd95-4366-b9a3-7ac35bb1f749" path="/var/lib/kubelet/pods/d2818938-bd95-4366-b9a3-7ac35bb1f749/volumes"
Jul 14 01:42:10.820571 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:10.820537    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:16.455028 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:16.454972    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:16.515698 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:16.515664    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:21.483607 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:21.483574    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:21.716226 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:21.716195    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:22.038347 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.038293    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:22.314393 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.314311    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"]
Jul 14 01:42:22.314648 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.314635    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2818938-bd95-4366-b9a3-7ac35bb1f749" containerName="maas-api"
Jul 14 01:42:22.314701 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.314650    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2818938-bd95-4366-b9a3-7ac35bb1f749" containerName="maas-api"
Jul 14 01:42:22.314701 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.314664    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f78ae992-8b27-4368-b38a-51c6e26a903a" containerName="maas-api"
Jul 14 01:42:22.314701 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.314670    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f78ae992-8b27-4368-b38a-51c6e26a903a" containerName="maas-api"
Jul 14 01:42:22.314701 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.314678    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" containerName="istio-proxy"
Jul 14 01:42:22.314701 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.314684    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" containerName="istio-proxy"
Jul 14 01:42:22.314701 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.314699    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" containerName="istio-proxy"
Jul 14 01:42:22.314898 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.314704    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" containerName="istio-proxy"
Jul 14 01:42:22.314898 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.314758    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f78ae992-8b27-4368-b38a-51c6e26a903a" containerName="maas-api"
Jul 14 01:42:22.314898 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.314767    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" containerName="istio-proxy"
Jul 14 01:42:22.314898 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.314773    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2818938-bd95-4366-b9a3-7ac35bb1f749" containerName="maas-api"
Jul 14 01:42:22.314898 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.314780    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b53a4ed-6863-4bf4-bebd-3f265fdf2dce" containerName="istio-proxy"
Jul 14 01:42:22.319033 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.319011    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.322147 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.322119    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-597948-openshift-default-dockercfg-sjqt5\""
Jul 14 01:42:22.332747 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.332720    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"]
Jul 14 01:42:22.334368 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.334337    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:22.503216 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.503172    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-workload-certs\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.503216 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.503218    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istiod-ca-cert\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.503724 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.503271    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-workload-socket\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.503724 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.503290    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-envoy\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.503724 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.503305    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-podinfo\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.503724 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.503406    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-token\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.503724 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.503473    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnh72\" (UniqueName: \"kubernetes.io/projected/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-kube-api-access-rnh72\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.503724 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.503505    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-data\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.503724 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.503540    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-credential-socket\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.509913 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.509883    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:22.605040 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.604920    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-workload-certs\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.605040 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.604984    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istiod-ca-cert\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.605040 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.605014    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-workload-socket\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.605040 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.605033    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-envoy\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.605040 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.605048    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-podinfo\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.605408 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.605095    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-token\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.605408 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.605142    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnh72\" (UniqueName: \"kubernetes.io/projected/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-kube-api-access-rnh72\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.605408 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.605171    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-data\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.605408 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.605207    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-credential-socket\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.605647 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.605399    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-workload-certs\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.605647 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.605517    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-data\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.605647 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.605589    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-credential-socket\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.605833 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.605661    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-workload-socket\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.605973 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.605934    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istiod-ca-cert\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.607513 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.607490    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-envoy\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.607665 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.607647    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-podinfo\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.612289 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.612263    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-token\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.612490 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.612470    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnh72\" (UniqueName: \"kubernetes.io/projected/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-kube-api-access-rnh72\") pod \"e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") " pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.636339 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.636307    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:22.779842 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.779805    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"]
Jul 14 01:42:22.781771 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:42:22.781738    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b8c525_42e6_46d0_85f2_7d9a2c0cdf08.slice/crio-7911a8a9f1ddbec6fc36b2f31d6bf37cf1afa0b004466087f4f6faea7761968a WatchSource:0}: Error finding container 7911a8a9f1ddbec6fc36b2f31d6bf37cf1afa0b004466087f4f6faea7761968a: Status 404 returned error can't find the container with id 7911a8a9f1ddbec6fc36b2f31d6bf37cf1afa0b004466087f4f6faea7761968a
Jul 14 01:42:22.783896 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.783866    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:42:22.783982 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.783931    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:42:22.783982 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:22.783978    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:42:23.704289 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:23.704252    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg" event={"ID":"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08","Type":"ContainerStarted","Data":"4a4e02885167f0187f28a63bc0205c4475ce764bae4c20a9d340aee20d45f922"}
Jul 14 01:42:23.704710 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:23.704307    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg" event={"ID":"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08","Type":"ContainerStarted","Data":"7911a8a9f1ddbec6fc36b2f31d6bf37cf1afa0b004466087f4f6faea7761968a"}
Jul 14 01:42:23.724166 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:23.724113    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg" podStartSLOduration=1.724096957 podStartE2EDuration="1.724096957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:42:23.722813859 +0000 UTC m=+2573.287222927" watchObservedRunningTime="2026-07-14 01:42:23.724096957 +0000 UTC m=+2573.288506011"
Jul 14 01:42:24.636608 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:24.636572    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:24.641512 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:24.641490    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:24.709576 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:24.709539    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:24.710584 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:24.710563    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:25.646819 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.646778    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:25.715947 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.715918    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"]
Jul 14 01:42:25.719997 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.719970    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"
Jul 14 01:42:25.723060 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.723023    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-597948-serving-cert\""
Jul 14 01:42:25.723227 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.723138    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8hlqx\""
Jul 14 01:42:25.723227 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.723139    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:42:25.723582 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.723561    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:42:25.735367 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.735332    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"]
Jul 14 01:42:25.742834 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.742756    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:25.807162 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.807129    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:25.836599 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.836560    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc2ms\" (UniqueName: \"kubernetes.io/projected/5f1fccd1-8cef-4160-8027-675d25d0ef5e-kube-api-access-lc2ms\") pod \"maas-api-e2e-api-a-597948-57fddbc7bc-k95gk\" (UID: \"5f1fccd1-8cef-4160-8027-675d25d0ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"
Jul 14 01:42:25.836913 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.836882    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1fccd1-8cef-4160-8027-675d25d0ef5e-openshift-service-ca\") pod \"maas-api-e2e-api-a-597948-57fddbc7bc-k95gk\" (UID: \"5f1fccd1-8cef-4160-8027-675d25d0ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"
Jul 14 01:42:25.837054 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.836929    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1fccd1-8cef-4160-8027-675d25d0ef5e-maas-api-tls\") pod \"maas-api-e2e-api-a-597948-57fddbc7bc-k95gk\" (UID: \"5f1fccd1-8cef-4160-8027-675d25d0ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"
Jul 14 01:42:25.911607 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.911521    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:25.937554 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.937519    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc2ms\" (UniqueName: \"kubernetes.io/projected/5f1fccd1-8cef-4160-8027-675d25d0ef5e-kube-api-access-lc2ms\") pod \"maas-api-e2e-api-a-597948-57fddbc7bc-k95gk\" (UID: \"5f1fccd1-8cef-4160-8027-675d25d0ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"
Jul 14 01:42:25.937757 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.937567    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1fccd1-8cef-4160-8027-675d25d0ef5e-openshift-service-ca\") pod \"maas-api-e2e-api-a-597948-57fddbc7bc-k95gk\" (UID: \"5f1fccd1-8cef-4160-8027-675d25d0ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"
Jul 14 01:42:25.937757 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.937738    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1fccd1-8cef-4160-8027-675d25d0ef5e-maas-api-tls\") pod \"maas-api-e2e-api-a-597948-57fddbc7bc-k95gk\" (UID: \"5f1fccd1-8cef-4160-8027-675d25d0ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"
Jul 14 01:42:25.938351 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.938329    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1fccd1-8cef-4160-8027-675d25d0ef5e-openshift-service-ca\") pod \"maas-api-e2e-api-a-597948-57fddbc7bc-k95gk\" (UID: \"5f1fccd1-8cef-4160-8027-675d25d0ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"
Jul 14 01:42:25.940407 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.940387    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1fccd1-8cef-4160-8027-675d25d0ef5e-maas-api-tls\") pod \"maas-api-e2e-api-a-597948-57fddbc7bc-k95gk\" (UID: \"5f1fccd1-8cef-4160-8027-675d25d0ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"
Jul 14 01:42:25.945537 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:25.945511    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc2ms\" (UniqueName: \"kubernetes.io/projected/5f1fccd1-8cef-4160-8027-675d25d0ef5e-kube-api-access-lc2ms\") pod \"maas-api-e2e-api-a-597948-57fddbc7bc-k95gk\" (UID: \"5f1fccd1-8cef-4160-8027-675d25d0ef5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"
Jul 14 01:42:26.035031 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:26.034995    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"
Jul 14 01:42:26.217718 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:26.217688    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"]
Jul 14 01:42:26.219364 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:42:26.219328    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1fccd1_8cef_4160_8027_675d25d0ef5e.slice/crio-b3213db1b31a851e98e6b7c0c049837f9aacbf87e76e0c963eb725e5e33fa901 WatchSource:0}: Error finding container b3213db1b31a851e98e6b7c0c049837f9aacbf87e76e0c963eb725e5e33fa901: Status 404 returned error can't find the container with id b3213db1b31a851e98e6b7c0c049837f9aacbf87e76e0c963eb725e5e33fa901
Jul 14 01:42:26.220774 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:26.220753    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 01:42:26.395263 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:26.395234    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:26.496471 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:26.496386    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:26.722502 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:26.722463    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk" event={"ID":"5f1fccd1-8cef-4160-8027-675d25d0ef5e","Type":"ContainerStarted","Data":"b3213db1b31a851e98e6b7c0c049837f9aacbf87e76e0c963eb725e5e33fa901"}
Jul 14 01:42:27.729032 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:27.728992    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk" event={"ID":"5f1fccd1-8cef-4160-8027-675d25d0ef5e","Type":"ContainerStarted","Data":"21b96c41365c9a1c921202ac08ec344a6e711f64f3f7771b47971167e9601985"}
Jul 14 01:42:27.729419 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:27.729208    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"
Jul 14 01:42:27.747401 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:27.747348    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk" podStartSLOduration=2.272112284 podStartE2EDuration="2.747331295s" totalImagesPullingTime="475.219011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:42:27.744827228 +0000 UTC m=+2577.309236282" watchObservedRunningTime="2026-07-14 01:42:27.747331295 +0000 UTC m=+2577.311740383"
Jul 14 01:42:30.130258 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:30.130226    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:30.565818 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:30.565725    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:30.865553 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:30.865458    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:31.156505 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:31.156469    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:33.747402 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:33.747364    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"
Jul 14 01:42:34.080524 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.080431    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:34.165273 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.165239    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:34.167276 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.167249    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"]
Jul 14 01:42:34.172461 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.172439    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:42:34.174795 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.174775    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-7b8e5a-serving-cert\""
Jul 14 01:42:34.181413 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.181384    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"]
Jul 14 01:42:34.212182 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.212148    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42dd72e1-2c7e-404d-a321-b05c1a76ea41-maas-api-tls\") pod \"maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks\" (UID: \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:42:34.212360 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.212264    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42dd72e1-2c7e-404d-a321-b05c1a76ea41-openshift-service-ca\") pod \"maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks\" (UID: \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:42:34.212360 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.212286    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzvgb\" (UniqueName: \"kubernetes.io/projected/42dd72e1-2c7e-404d-a321-b05c1a76ea41-kube-api-access-bzvgb\") pod \"maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks\" (UID: \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:42:34.230360 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.230325    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:34.313805 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.313760    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42dd72e1-2c7e-404d-a321-b05c1a76ea41-openshift-service-ca\") pod \"maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks\" (UID: \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:42:34.313805 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.313802    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzvgb\" (UniqueName: \"kubernetes.io/projected/42dd72e1-2c7e-404d-a321-b05c1a76ea41-kube-api-access-bzvgb\") pod \"maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks\" (UID: \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:42:34.314069 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.313922    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42dd72e1-2c7e-404d-a321-b05c1a76ea41-maas-api-tls\") pod \"maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks\" (UID: \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:42:34.314110 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:42:34.314066    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-serving-cert: secret "maas-api-e2e-api-b-7b8e5a-serving-cert" not found
Jul 14 01:42:34.314152 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:42:34.314141    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42dd72e1-2c7e-404d-a321-b05c1a76ea41-maas-api-tls podName:42dd72e1-2c7e-404d-a321-b05c1a76ea41 nodeName:}" failed. No retries permitted until 2026-07-14 01:42:34.814125588 +0000 UTC m=+2584.378534621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42dd72e1-2c7e-404d-a321-b05c1a76ea41-maas-api-tls") pod "maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks" (UID: "42dd72e1-2c7e-404d-a321-b05c1a76ea41") : secret "maas-api-e2e-api-b-7b8e5a-serving-cert" not found
Jul 14 01:42:34.314488 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.314469    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42dd72e1-2c7e-404d-a321-b05c1a76ea41-openshift-service-ca\") pod \"maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks\" (UID: \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:42:34.323386 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.323351    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzvgb\" (UniqueName: \"kubernetes.io/projected/42dd72e1-2c7e-404d-a321-b05c1a76ea41-kube-api-access-bzvgb\") pod \"maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks\" (UID: \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:42:34.546501 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.546452    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:34.723797 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.723767    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:34.817948 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.817849    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42dd72e1-2c7e-404d-a321-b05c1a76ea41-maas-api-tls\") pod \"maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks\" (UID: \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:42:34.820526 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.820498    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42dd72e1-2c7e-404d-a321-b05c1a76ea41-maas-api-tls\") pod \"maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks\" (UID: \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:42:34.825438 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:34.825414    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:35.085999 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:35.085882    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:42:35.257169 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:42:35.257135    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42dd72e1_2c7e_404d_a321_b05c1a76ea41.slice/crio-7d3e50081f5b4cea3709e1d826948a1a44b8caf3cfb233ecd4199e108f706053 WatchSource:0}: Error finding container 7d3e50081f5b4cea3709e1d826948a1a44b8caf3cfb233ecd4199e108f706053: Status 404 returned error can't find the container with id 7d3e50081f5b4cea3709e1d826948a1a44b8caf3cfb233ecd4199e108f706053
Jul 14 01:42:35.257302 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:35.257249    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"]
Jul 14 01:42:35.779277 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:35.779239    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks" event={"ID":"42dd72e1-2c7e-404d-a321-b05c1a76ea41","Type":"ContainerStarted","Data":"52f24848ea5745f4d87b9af91667cd1f56d14faf82cce03099c86d22af909653"}
Jul 14 01:42:35.779492 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:35.779303    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks" event={"ID":"42dd72e1-2c7e-404d-a321-b05c1a76ea41","Type":"ContainerStarted","Data":"7d3e50081f5b4cea3709e1d826948a1a44b8caf3cfb233ecd4199e108f706053"}
Jul 14 01:42:35.779492 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:35.779356    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:42:35.796187 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:35.796059    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks" podStartSLOduration=1.508792044 podStartE2EDuration="1.796033716s" totalImagesPullingTime="287.241672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:42:35.793060016 +0000 UTC m=+2585.357469073" watchObservedRunningTime="2026-07-14 01:42:35.796033716 +0000 UTC m=+2585.360442782"
Jul 14 01:42:38.611768 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:38.611731    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:38.801923 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:38.801892    2567 generic.go:356] "Generic (PLEG): container finished" podID="c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" containerID="4a4e02885167f0187f28a63bc0205c4475ce764bae4c20a9d340aee20d45f922" exitCode=0
Jul 14 01:42:38.802107 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:38.801984    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg" event={"ID":"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08","Type":"ContainerDied","Data":"4a4e02885167f0187f28a63bc0205c4475ce764bae4c20a9d340aee20d45f922"}
Jul 14 01:42:38.802411 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:38.802397    2567 scope.go:117] "RemoveContainer" containerID="4a4e02885167f0187f28a63bc0205c4475ce764bae4c20a9d340aee20d45f922"
Jul 14 01:42:38.803529 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:38.803496    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:42:38.803610 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:38.803591    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:42:38.803656 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:38.803639    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:42:39.710374 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:39.710307    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:39.809487 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:39.809446    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg" event={"ID":"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08","Type":"ContainerStarted","Data":"361a5b6e5f47c986ccdc26a9eb88c96c2329ec0596e45e9bc499780de18e0a5b"}
Jul 14 01:42:40.637235 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:40.637136    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:40.638753 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:40.638713    2567 patch_prober.go:28] interesting pod/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 14 01:42:40.638918 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:40.638790    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg" podUID="c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 14 01:42:41.637540 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:41.637499    2567 patch_prober.go:28] interesting pod/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 14 01:42:41.638097 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:41.637586    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg" podUID="c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 14 01:42:41.791946 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:41.791917    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:42:42.637114 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:42.637075    2567 patch_prober.go:28] interesting pod/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 14 01:42:42.637855 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:42.637409    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg" podUID="c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 14 01:42:43.637828 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:43.637789    2567 patch_prober.go:28] interesting pod/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 14 01:42:43.638161 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:43.638129    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg" podUID="c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 14 01:42:44.641089 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:44.641043    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:44.641485 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:44.641435    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:44.642320 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:44.642305    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:42:47.099595 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:47.099562    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:47.200191 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:47.200154    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:55.568821 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:55.568743    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:55.671813 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:55.671781    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:57.091524 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.091493    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:57.174969 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.174924    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:57.315142 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.313806    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"]
Jul 14 01:42:57.315142 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.314140    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk" podUID="5f1fccd1-8cef-4160-8027-675d25d0ef5e" containerName="maas-api" containerID="cri-o://21b96c41365c9a1c921202ac08ec344a6e711f64f3f7771b47971167e9601985" gracePeriod=30
Jul 14 01:42:57.335263 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.335229    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:42:57.583038 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.583012    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"
Jul 14 01:42:57.720696 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.720586    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1fccd1-8cef-4160-8027-675d25d0ef5e-openshift-service-ca\") pod \"5f1fccd1-8cef-4160-8027-675d25d0ef5e\" (UID: \"5f1fccd1-8cef-4160-8027-675d25d0ef5e\") "
Jul 14 01:42:57.720696 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.720657    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1fccd1-8cef-4160-8027-675d25d0ef5e-maas-api-tls\") pod \"5f1fccd1-8cef-4160-8027-675d25d0ef5e\" (UID: \"5f1fccd1-8cef-4160-8027-675d25d0ef5e\") "
Jul 14 01:42:57.720696 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.720696    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc2ms\" (UniqueName: \"kubernetes.io/projected/5f1fccd1-8cef-4160-8027-675d25d0ef5e-kube-api-access-lc2ms\") pod \"5f1fccd1-8cef-4160-8027-675d25d0ef5e\" (UID: \"5f1fccd1-8cef-4160-8027-675d25d0ef5e\") "
Jul 14 01:42:57.721004 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.720981    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f1fccd1-8cef-4160-8027-675d25d0ef5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f1fccd1-8cef-4160-8027-675d25d0ef5e" (UID: "5f1fccd1-8cef-4160-8027-675d25d0ef5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:42:57.723067 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.723043    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f1fccd1-8cef-4160-8027-675d25d0ef5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f1fccd1-8cef-4160-8027-675d25d0ef5e" (UID: "5f1fccd1-8cef-4160-8027-675d25d0ef5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:42:57.723147 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.723081    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1fccd1-8cef-4160-8027-675d25d0ef5e-kube-api-access-lc2ms" (OuterVolumeSpecName: "kube-api-access-lc2ms") pod "5f1fccd1-8cef-4160-8027-675d25d0ef5e" (UID: "5f1fccd1-8cef-4160-8027-675d25d0ef5e"). InnerVolumeSpecName "kube-api-access-lc2ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:42:57.822073 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.822018    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc2ms\" (UniqueName: \"kubernetes.io/projected/5f1fccd1-8cef-4160-8027-675d25d0ef5e-kube-api-access-lc2ms\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:42:57.822073 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.822058    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1fccd1-8cef-4160-8027-675d25d0ef5e-openshift-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:42:57.822073 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.822071    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1fccd1-8cef-4160-8027-675d25d0ef5e-maas-api-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:42:57.914269 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.914236    2567 generic.go:356] "Generic (PLEG): container finished" podID="5f1fccd1-8cef-4160-8027-675d25d0ef5e" containerID="21b96c41365c9a1c921202ac08ec344a6e711f64f3f7771b47971167e9601985" exitCode=0
Jul 14 01:42:57.914452 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.914281    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"
Jul 14 01:42:57.914452 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.914330    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk" event={"ID":"5f1fccd1-8cef-4160-8027-675d25d0ef5e","Type":"ContainerDied","Data":"21b96c41365c9a1c921202ac08ec344a6e711f64f3f7771b47971167e9601985"}
Jul 14 01:42:57.914452 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.914387    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk" event={"ID":"5f1fccd1-8cef-4160-8027-675d25d0ef5e","Type":"ContainerDied","Data":"b3213db1b31a851e98e6b7c0c049837f9aacbf87e76e0c963eb725e5e33fa901"}
Jul 14 01:42:57.914452 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.914410    2567 scope.go:117] "RemoveContainer" containerID="21b96c41365c9a1c921202ac08ec344a6e711f64f3f7771b47971167e9601985"
Jul 14 01:42:57.924522 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.924502    2567 scope.go:117] "RemoveContainer" containerID="21b96c41365c9a1c921202ac08ec344a6e711f64f3f7771b47971167e9601985"
Jul 14 01:42:57.924899 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:42:57.924871    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b96c41365c9a1c921202ac08ec344a6e711f64f3f7771b47971167e9601985\": container with ID starting with 21b96c41365c9a1c921202ac08ec344a6e711f64f3f7771b47971167e9601985 not found: ID does not exist" containerID="21b96c41365c9a1c921202ac08ec344a6e711f64f3f7771b47971167e9601985"
Jul 14 01:42:57.924974 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.924915    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b96c41365c9a1c921202ac08ec344a6e711f64f3f7771b47971167e9601985"} err="failed to get container status \"21b96c41365c9a1c921202ac08ec344a6e711f64f3f7771b47971167e9601985\": rpc error: code = NotFound desc = could not find container \"21b96c41365c9a1c921202ac08ec344a6e711f64f3f7771b47971167e9601985\": container with ID starting with 21b96c41365c9a1c921202ac08ec344a6e711f64f3f7771b47971167e9601985 not found: ID does not exist"
Jul 14 01:42:57.935154 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.935125    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"]
Jul 14 01:42:57.938532 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:57.938504    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-597948-57fddbc7bc-k95gk"]
Jul 14 01:42:58.980121 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:42:58.980084    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f1fccd1-8cef-4160-8027-675d25d0ef5e" path="/var/lib/kubelet/pods/5f1fccd1-8cef-4160-8027-675d25d0ef5e/volumes"
Jul 14 01:43:01.671919 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:01.671882    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:07.407082 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:07.407047    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:07.508272 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:07.508231    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:12.334624 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:12.334586    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"]
Jul 14 01:43:12.335026 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:12.334855    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg" podUID="c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" containerName="istio-proxy" containerID="cri-o://361a5b6e5f47c986ccdc26a9eb88c96c2329ec0596e45e9bc499780de18e0a5b" gracePeriod=30
Jul 14 01:43:12.350556 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:12.350523    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:12.593059 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:12.592945    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:12.733085 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:12.733040    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:13.291968 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:13.291933    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:13.392786 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:13.392752    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:14.644299 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:14.644256    2567 patch_prober.go:28] interesting pod/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 01:43:14.644758 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:14.644339    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg" podUID="c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 01:43:17.599876 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.599795    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:43:17.706689 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.706653    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-envoy\") pod \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") "
Jul 14 01:43:17.706896 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.706709    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-podinfo\") pod \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") "
Jul 14 01:43:17.706896 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.706751    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-token\") pod \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") "
Jul 14 01:43:17.706896 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.706776    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-data\") pod \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") "
Jul 14 01:43:17.706896 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.706808    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-credential-socket\") pod \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") "
Jul 14 01:43:17.706896 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.706835    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnh72\" (UniqueName: \"kubernetes.io/projected/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-kube-api-access-rnh72\") pod \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") "
Jul 14 01:43:17.706896 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.706873    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-workload-certs\") pod \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") "
Jul 14 01:43:17.706896 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.706889    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istiod-ca-cert\") pod \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") "
Jul 14 01:43:17.707308 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.706911    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-workload-socket\") pod \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\" (UID: \"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08\") "
Jul 14 01:43:17.707308 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.707248    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" (UID: "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:43:17.707412 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.707338    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" (UID: "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:43:17.707614 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.707485    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" (UID: "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:43:17.707614 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.707553    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" (UID: "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:43:17.708106 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.708073    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-data" (OuterVolumeSpecName: "istio-data") pod "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" (UID: "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:43:17.709271 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.709238    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" (UID: "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:43:17.709271 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.709251    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-token" (OuterVolumeSpecName: "istio-token") pod "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" (UID: "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:43:17.709448 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.709289    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" (UID: "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 01:43:17.709533 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.709514    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-kube-api-access-rnh72" (OuterVolumeSpecName: "kube-api-access-rnh72") pod "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" (UID: "c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08"). InnerVolumeSpecName "kube-api-access-rnh72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:43:17.807598 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.807554    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-token\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:43:17.807598 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.807588    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-data\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:43:17.807598 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.807597    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-credential-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:43:17.807598 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.807608    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnh72\" (UniqueName: \"kubernetes.io/projected/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-kube-api-access-rnh72\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:43:17.807877 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.807619    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-workload-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:43:17.807877 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.807628    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istiod-ca-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:43:17.807877 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.807635    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-workload-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:43:17.807877 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.807643    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-envoy\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:43:17.807877 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:17.807651    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08-istio-podinfo\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:43:18.019137 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:18.019097    2567 generic.go:356] "Generic (PLEG): container finished" podID="c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" containerID="361a5b6e5f47c986ccdc26a9eb88c96c2329ec0596e45e9bc499780de18e0a5b" exitCode=0
Jul 14 01:43:18.019315 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:18.019155    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"
Jul 14 01:43:18.019315 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:18.019192    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg" event={"ID":"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08","Type":"ContainerDied","Data":"361a5b6e5f47c986ccdc26a9eb88c96c2329ec0596e45e9bc499780de18e0a5b"}
Jul 14 01:43:18.019315 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:18.019249    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg" event={"ID":"c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08","Type":"ContainerDied","Data":"7911a8a9f1ddbec6fc36b2f31d6bf37cf1afa0b004466087f4f6faea7761968a"}
Jul 14 01:43:18.019315 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:18.019275    2567 scope.go:117] "RemoveContainer" containerID="361a5b6e5f47c986ccdc26a9eb88c96c2329ec0596e45e9bc499780de18e0a5b"
Jul 14 01:43:18.029077 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:18.029056    2567 scope.go:117] "RemoveContainer" containerID="4a4e02885167f0187f28a63bc0205c4475ce764bae4c20a9d340aee20d45f922"
Jul 14 01:43:18.037997 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:18.037976    2567 scope.go:117] "RemoveContainer" containerID="361a5b6e5f47c986ccdc26a9eb88c96c2329ec0596e45e9bc499780de18e0a5b"
Jul 14 01:43:18.038299 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:43:18.038280    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361a5b6e5f47c986ccdc26a9eb88c96c2329ec0596e45e9bc499780de18e0a5b\": container with ID starting with 361a5b6e5f47c986ccdc26a9eb88c96c2329ec0596e45e9bc499780de18e0a5b not found: ID does not exist" containerID="361a5b6e5f47c986ccdc26a9eb88c96c2329ec0596e45e9bc499780de18e0a5b"
Jul 14 01:43:18.038379 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:18.038308    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361a5b6e5f47c986ccdc26a9eb88c96c2329ec0596e45e9bc499780de18e0a5b"} err="failed to get container status \"361a5b6e5f47c986ccdc26a9eb88c96c2329ec0596e45e9bc499780de18e0a5b\": rpc error: code = NotFound desc = could not find container \"361a5b6e5f47c986ccdc26a9eb88c96c2329ec0596e45e9bc499780de18e0a5b\": container with ID starting with 361a5b6e5f47c986ccdc26a9eb88c96c2329ec0596e45e9bc499780de18e0a5b not found: ID does not exist"
Jul 14 01:43:18.038379 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:18.038328    2567 scope.go:117] "RemoveContainer" containerID="4a4e02885167f0187f28a63bc0205c4475ce764bae4c20a9d340aee20d45f922"
Jul 14 01:43:18.038601 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:43:18.038581    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4e02885167f0187f28a63bc0205c4475ce764bae4c20a9d340aee20d45f922\": container with ID starting with 4a4e02885167f0187f28a63bc0205c4475ce764bae4c20a9d340aee20d45f922 not found: ID does not exist" containerID="4a4e02885167f0187f28a63bc0205c4475ce764bae4c20a9d340aee20d45f922"
Jul 14 01:43:18.038641 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:18.038608    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4e02885167f0187f28a63bc0205c4475ce764bae4c20a9d340aee20d45f922"} err="failed to get container status \"4a4e02885167f0187f28a63bc0205c4475ce764bae4c20a9d340aee20d45f922\": rpc error: code = NotFound desc = could not find container \"4a4e02885167f0187f28a63bc0205c4475ce764bae4c20a9d340aee20d45f922\": container with ID starting with 4a4e02885167f0187f28a63bc0205c4475ce764bae4c20a9d340aee20d45f922 not found: ID does not exist"
Jul 14 01:43:18.043861 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:18.043824    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"]
Jul 14 01:43:18.049153 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:18.049125    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-597948-openshift-default-55c76cdb59-xbzvg"]
Jul 14 01:43:18.975754 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:18.975715    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" path="/var/lib/kubelet/pods/c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08/volumes"
Jul 14 01:43:23.144294 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:23.144251    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"]
Jul 14 01:43:23.144812 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:23.144500    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks" podUID="42dd72e1-2c7e-404d-a321-b05c1a76ea41" containerName="maas-api" containerID="cri-o://52f24848ea5745f4d87b9af91667cd1f56d14faf82cce03099c86d22af909653" gracePeriod=30
Jul 14 01:43:23.184835 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:23.184805    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:23.398236 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:23.398169    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:43:23.555676 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:23.555635    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzvgb\" (UniqueName: \"kubernetes.io/projected/42dd72e1-2c7e-404d-a321-b05c1a76ea41-kube-api-access-bzvgb\") pod \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\" (UID: \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\") "
Jul 14 01:43:23.555676 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:23.555677    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42dd72e1-2c7e-404d-a321-b05c1a76ea41-maas-api-tls\") pod \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\" (UID: \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\") "
Jul 14 01:43:23.555945 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:23.555720    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42dd72e1-2c7e-404d-a321-b05c1a76ea41-openshift-service-ca\") pod \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\" (UID: \"42dd72e1-2c7e-404d-a321-b05c1a76ea41\") "
Jul 14 01:43:23.556198 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:23.556167    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42dd72e1-2c7e-404d-a321-b05c1a76ea41-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42dd72e1-2c7e-404d-a321-b05c1a76ea41" (UID: "42dd72e1-2c7e-404d-a321-b05c1a76ea41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:43:23.557922 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:23.557897    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42dd72e1-2c7e-404d-a321-b05c1a76ea41-kube-api-access-bzvgb" (OuterVolumeSpecName: "kube-api-access-bzvgb") pod "42dd72e1-2c7e-404d-a321-b05c1a76ea41" (UID: "42dd72e1-2c7e-404d-a321-b05c1a76ea41"). InnerVolumeSpecName "kube-api-access-bzvgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:43:23.558024 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:23.557917    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42dd72e1-2c7e-404d-a321-b05c1a76ea41-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42dd72e1-2c7e-404d-a321-b05c1a76ea41" (UID: "42dd72e1-2c7e-404d-a321-b05c1a76ea41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:43:23.656890 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:23.656790    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42dd72e1-2c7e-404d-a321-b05c1a76ea41-openshift-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:43:23.656890 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:23.656824    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzvgb\" (UniqueName: \"kubernetes.io/projected/42dd72e1-2c7e-404d-a321-b05c1a76ea41-kube-api-access-bzvgb\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:43:23.656890 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:23.656837    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42dd72e1-2c7e-404d-a321-b05c1a76ea41-maas-api-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:43:24.054167 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:24.054068    2567 generic.go:356] "Generic (PLEG): container finished" podID="42dd72e1-2c7e-404d-a321-b05c1a76ea41" containerID="52f24848ea5745f4d87b9af91667cd1f56d14faf82cce03099c86d22af909653" exitCode=0
Jul 14 01:43:24.054167 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:24.054113    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"
Jul 14 01:43:24.054167 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:24.054153    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks" event={"ID":"42dd72e1-2c7e-404d-a321-b05c1a76ea41","Type":"ContainerDied","Data":"52f24848ea5745f4d87b9af91667cd1f56d14faf82cce03099c86d22af909653"}
Jul 14 01:43:24.054418 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:24.054210    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks" event={"ID":"42dd72e1-2c7e-404d-a321-b05c1a76ea41","Type":"ContainerDied","Data":"7d3e50081f5b4cea3709e1d826948a1a44b8caf3cfb233ecd4199e108f706053"}
Jul 14 01:43:24.054418 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:24.054232    2567 scope.go:117] "RemoveContainer" containerID="52f24848ea5745f4d87b9af91667cd1f56d14faf82cce03099c86d22af909653"
Jul 14 01:43:24.063657 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:24.063636    2567 scope.go:117] "RemoveContainer" containerID="52f24848ea5745f4d87b9af91667cd1f56d14faf82cce03099c86d22af909653"
Jul 14 01:43:24.063989 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:43:24.063944    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f24848ea5745f4d87b9af91667cd1f56d14faf82cce03099c86d22af909653\": container with ID starting with 52f24848ea5745f4d87b9af91667cd1f56d14faf82cce03099c86d22af909653 not found: ID does not exist" containerID="52f24848ea5745f4d87b9af91667cd1f56d14faf82cce03099c86d22af909653"
Jul 14 01:43:24.064076 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:24.064002    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f24848ea5745f4d87b9af91667cd1f56d14faf82cce03099c86d22af909653"} err="failed to get container status \"52f24848ea5745f4d87b9af91667cd1f56d14faf82cce03099c86d22af909653\": rpc error: code = NotFound desc = could not find container \"52f24848ea5745f4d87b9af91667cd1f56d14faf82cce03099c86d22af909653\": container with ID starting with 52f24848ea5745f4d87b9af91667cd1f56d14faf82cce03099c86d22af909653 not found: ID does not exist"
Jul 14 01:43:24.074519 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:24.074488    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"]
Jul 14 01:43:24.078335 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:24.078305    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-7b8e5a-59566c8554-6cwks"]
Jul 14 01:43:24.974370 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:24.974333    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42dd72e1-2c7e-404d-a321-b05c1a76ea41" path="/var/lib/kubelet/pods/42dd72e1-2c7e-404d-a321-b05c1a76ea41/volumes"
Jul 14 01:43:27.808373 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:27.808334    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:33.705642 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:33.705608    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:33.807802 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:33.807766    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:38.539606 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:38.539554    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:38.795216 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:38.795129    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:39.157144 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.157096    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:39.436350 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.436256    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"]
Jul 14 01:43:39.436670 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.436651    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" containerName="istio-proxy"
Jul 14 01:43:39.436752 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.436672    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" containerName="istio-proxy"
Jul 14 01:43:39.436752 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.436684    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42dd72e1-2c7e-404d-a321-b05c1a76ea41" containerName="maas-api"
Jul 14 01:43:39.436752 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.436691    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="42dd72e1-2c7e-404d-a321-b05c1a76ea41" containerName="maas-api"
Jul 14 01:43:39.436752 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.436699    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" containerName="istio-proxy"
Jul 14 01:43:39.436752 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.436708    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" containerName="istio-proxy"
Jul 14 01:43:39.436752 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.436734    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f1fccd1-8cef-4160-8027-675d25d0ef5e" containerName="maas-api"
Jul 14 01:43:39.436752 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.436742    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f1fccd1-8cef-4160-8027-675d25d0ef5e" containerName="maas-api"
Jul 14 01:43:39.437115 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.436853    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f1fccd1-8cef-4160-8027-675d25d0ef5e" containerName="maas-api"
Jul 14 01:43:39.437115 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.436867    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" containerName="istio-proxy"
Jul 14 01:43:39.437115 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.436878    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="42dd72e1-2c7e-404d-a321-b05c1a76ea41" containerName="maas-api"
Jul 14 01:43:39.437115 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.436887    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5b8c525-42e6-46d0-85f2-7d9a2c0cdf08" containerName="istio-proxy"
Jul 14 01:43:39.439991 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.439944    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.442349 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.442326    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-99e8d0-openshift-default-dockercfg-hz9xw\""
Jul 14 01:43:39.452347 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.452321    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"]
Jul 14 01:43:39.470918 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.470884    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:39.480862 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.479102    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istiod-ca-cert\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.480862 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.479165    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-token\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.480862 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.479223    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5l4z\" (UniqueName: \"kubernetes.io/projected/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-kube-api-access-l5l4z\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.480862 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.479266    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-credential-socket\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.480862 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.479299    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-data\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.480862 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.479327    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-workload-socket\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.480862 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.479349    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-workload-certs\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.480862 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.479388    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-envoy\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.480862 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.479412    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-podinfo\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.580363 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.580324    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5l4z\" (UniqueName: \"kubernetes.io/projected/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-kube-api-access-l5l4z\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.580820 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.580378    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-credential-socket\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.580820 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.580414    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-data\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.580820 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.580442    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-workload-socket\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.580820 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.580468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-workload-certs\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.580820 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.580505    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-envoy\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.580820 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.580527    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-podinfo\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.580820 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.580570    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istiod-ca-cert\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.580820 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.580609    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-token\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.581276 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.580907    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-credential-socket\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.581276 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.580938    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-workload-certs\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.581276 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.581036    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-workload-socket\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.581276 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.581065    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-data\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.581494 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.581475    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istiod-ca-cert\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.583141 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.583120    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-envoy\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.583359 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.583339    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-podinfo\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.587453 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.587426    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-token\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.587453 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.587444    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5l4z\" (UniqueName: \"kubernetes.io/projected/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-kube-api-access-l5l4z\") pod \"e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") " pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.662179 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.662139    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:39.751003 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.750884    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:39.864933 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.864890    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:39.893492 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.893467    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"]
Jul 14 01:43:39.895582 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:43:39.895555    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf34f8c_b7ad_4e0c_8206_ba2bd55bc123.slice/crio-e35e3ded4d725833ebafc71673cda7969d6f81503ee825378938adb72bc003f6 WatchSource:0}: Error finding container e35e3ded4d725833ebafc71673cda7969d6f81503ee825378938adb72bc003f6: Status 404 returned error can't find the container with id e35e3ded4d725833ebafc71673cda7969d6f81503ee825378938adb72bc003f6
Jul 14 01:43:39.898167 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.898137    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:43:39.898231 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.898204    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:43:39.898277 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:39.898231    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:43:40.141416 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:40.141374    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq" event={"ID":"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123","Type":"ContainerStarted","Data":"a28a19d14984dcadeaf7621a3768018105a659feee19b7e280f6120c21df79eb"}
Jul 14 01:43:40.141628 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:40.141445    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq" event={"ID":"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123","Type":"ContainerStarted","Data":"e35e3ded4d725833ebafc71673cda7969d6f81503ee825378938adb72bc003f6"}
Jul 14 01:43:40.161504 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:40.161453    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq" podStartSLOduration=1.161433973 podStartE2EDuration="1.161433973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:43:40.158875302 +0000 UTC m=+2649.723284358" watchObservedRunningTime="2026-07-14 01:43:40.161433973 +0000 UTC m=+2649.725843027"
Jul 14 01:43:40.751521 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:40.751478    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:40.756850 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:40.756818    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:41.147367 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:41.147336    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:41.148383 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:41.148367    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:42.220975 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.220924    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:42.322443 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.322381    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:42.348688 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.348647    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"]
Jul 14 01:43:42.352480 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.352452    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"
Jul 14 01:43:42.354274 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.354248    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:43:42.354427 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.354327    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8hlqx\""
Jul 14 01:43:42.354816 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.354796    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-99e8d0-serving-cert\""
Jul 14 01:43:42.354816 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.354811    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:43:42.361813 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.361786    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"]
Jul 14 01:43:42.407287 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.407247    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27e93a0-3953-4d08-9fe7-e73f09b886b3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd\" (UID: \"a27e93a0-3953-4d08-9fe7-e73f09b886b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"
Jul 14 01:43:42.407287 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.407290    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqkz2\" (UniqueName: \"kubernetes.io/projected/a27e93a0-3953-4d08-9fe7-e73f09b886b3-kube-api-access-kqkz2\") pod \"maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd\" (UID: \"a27e93a0-3953-4d08-9fe7-e73f09b886b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"
Jul 14 01:43:42.407501 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.407314    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a27e93a0-3953-4d08-9fe7-e73f09b886b3-maas-api-tls\") pod \"maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd\" (UID: \"a27e93a0-3953-4d08-9fe7-e73f09b886b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"
Jul 14 01:43:42.419280 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.419250    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:42.508457 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.508343    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a27e93a0-3953-4d08-9fe7-e73f09b886b3-maas-api-tls\") pod \"maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd\" (UID: \"a27e93a0-3953-4d08-9fe7-e73f09b886b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"
Jul 14 01:43:42.508651 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.508468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27e93a0-3953-4d08-9fe7-e73f09b886b3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd\" (UID: \"a27e93a0-3953-4d08-9fe7-e73f09b886b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"
Jul 14 01:43:42.508651 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.508495    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqkz2\" (UniqueName: \"kubernetes.io/projected/a27e93a0-3953-4d08-9fe7-e73f09b886b3-kube-api-access-kqkz2\") pod \"maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd\" (UID: \"a27e93a0-3953-4d08-9fe7-e73f09b886b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"
Jul 14 01:43:42.509443 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.509418    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27e93a0-3953-4d08-9fe7-e73f09b886b3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd\" (UID: \"a27e93a0-3953-4d08-9fe7-e73f09b886b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"
Jul 14 01:43:42.511308 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.511283    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a27e93a0-3953-4d08-9fe7-e73f09b886b3-maas-api-tls\") pod \"maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd\" (UID: \"a27e93a0-3953-4d08-9fe7-e73f09b886b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"
Jul 14 01:43:42.517088 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.517055    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqkz2\" (UniqueName: \"kubernetes.io/projected/a27e93a0-3953-4d08-9fe7-e73f09b886b3-kube-api-access-kqkz2\") pod \"maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd\" (UID: \"a27e93a0-3953-4d08-9fe7-e73f09b886b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"
Jul 14 01:43:42.548600 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.548570    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:42.665666 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.665627    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"
Jul 14 01:43:42.811079 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.811050    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"]
Jul 14 01:43:42.813062 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:43:42.813029    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27e93a0_3953_4d08_9fe7_e73f09b886b3.slice/crio-4db2152cde851b46a2bd2574cb9c03938527f60e6234a8635845f9304e002652 WatchSource:0}: Error finding container 4db2152cde851b46a2bd2574cb9c03938527f60e6234a8635845f9304e002652: Status 404 returned error can't find the container with id 4db2152cde851b46a2bd2574cb9c03938527f60e6234a8635845f9304e002652
Jul 14 01:43:42.868475 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.868432    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:42.969404 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:42.969373    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:43.158904 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:43.158865    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd" event={"ID":"a27e93a0-3953-4d08-9fe7-e73f09b886b3","Type":"ContainerStarted","Data":"4db2152cde851b46a2bd2574cb9c03938527f60e6234a8635845f9304e002652"}
Jul 14 01:43:44.165113 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:44.165079    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd" event={"ID":"a27e93a0-3953-4d08-9fe7-e73f09b886b3","Type":"ContainerStarted","Data":"01dcdb6de01fcc857482e9da9bdd6cf389e0a38502b5e3bc2807debee8945982"}
Jul 14 01:43:44.165516 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:44.165178    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"
Jul 14 01:43:44.181677 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:44.181626    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd" podStartSLOduration=1.760905503 podStartE2EDuration="2.181607281s" totalImagesPullingTime="420.701778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:43:44.180173435 +0000 UTC m=+2653.744582489" watchObservedRunningTime="2026-07-14 01:43:44.181607281 +0000 UTC m=+2653.746016336"
Jul 14 01:43:46.742778 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:46.742731    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:47.080431 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:47.080331    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:47.392252 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:47.392217    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:47.612449 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:47.612411    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:50.176820 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.176790    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"
Jul 14 01:43:50.475192 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.475098    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:50.550852 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.550821    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:50.568381 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.566541    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"]
Jul 14 01:43:50.571058 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.571031    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"]
Jul 14 01:43:50.571208 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.571085    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"
Jul 14 01:43:50.573214 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.573147    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-e42ce1-serving-cert\""
Jul 14 01:43:50.611734 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.611703    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:50.683015 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.682977    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8v6l\" (UniqueName: \"kubernetes.io/projected/4d997dfc-d67d-4090-8de2-1486d6c0d44d-kube-api-access-m8v6l\") pod \"maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh\" (UID: \"4d997dfc-d67d-4090-8de2-1486d6c0d44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"
Jul 14 01:43:50.683187 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.683020    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d997dfc-d67d-4090-8de2-1486d6c0d44d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh\" (UID: \"4d997dfc-d67d-4090-8de2-1486d6c0d44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"
Jul 14 01:43:50.683187 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.683087    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d997dfc-d67d-4090-8de2-1486d6c0d44d-maas-api-tls\") pod \"maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh\" (UID: \"4d997dfc-d67d-4090-8de2-1486d6c0d44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"
Jul 14 01:43:50.784447 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.784353    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8v6l\" (UniqueName: \"kubernetes.io/projected/4d997dfc-d67d-4090-8de2-1486d6c0d44d-kube-api-access-m8v6l\") pod \"maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh\" (UID: \"4d997dfc-d67d-4090-8de2-1486d6c0d44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"
Jul 14 01:43:50.784447 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.784397    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d997dfc-d67d-4090-8de2-1486d6c0d44d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh\" (UID: \"4d997dfc-d67d-4090-8de2-1486d6c0d44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"
Jul 14 01:43:50.784447 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.784419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d997dfc-d67d-4090-8de2-1486d6c0d44d-maas-api-tls\") pod \"maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh\" (UID: \"4d997dfc-d67d-4090-8de2-1486d6c0d44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"
Jul 14 01:43:50.785118 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.785091    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d997dfc-d67d-4090-8de2-1486d6c0d44d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh\" (UID: \"4d997dfc-d67d-4090-8de2-1486d6c0d44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"
Jul 14 01:43:50.787077 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.787055    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d997dfc-d67d-4090-8de2-1486d6c0d44d-maas-api-tls\") pod \"maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh\" (UID: \"4d997dfc-d67d-4090-8de2-1486d6c0d44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"
Jul 14 01:43:50.797189 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.797161    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8v6l\" (UniqueName: \"kubernetes.io/projected/4d997dfc-d67d-4090-8de2-1486d6c0d44d-kube-api-access-m8v6l\") pod \"maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh\" (UID: \"4d997dfc-d67d-4090-8de2-1486d6c0d44d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"
Jul 14 01:43:50.809465 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.809438    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:50.884206 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:50.884172    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"
Jul 14 01:43:51.062916 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:51.062615    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"]
Jul 14 01:43:51.067633 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:43:51.067286    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d997dfc_d67d_4090_8de2_1486d6c0d44d.slice/crio-46ed5ee3b3b3442effddc3b6b560439612d70fe513cc25208c11df7cffb84a38 WatchSource:0}: Error finding container 46ed5ee3b3b3442effddc3b6b560439612d70fe513cc25208c11df7cffb84a38: Status 404 returned error can't find the container with id 46ed5ee3b3b3442effddc3b6b560439612d70fe513cc25208c11df7cffb84a38
Jul 14 01:43:51.157711 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:51.157542    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:51.208814 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:51.208780    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh" event={"ID":"4d997dfc-d67d-4090-8de2-1486d6c0d44d","Type":"ContainerStarted","Data":"46ed5ee3b3b3442effddc3b6b560439612d70fe513cc25208c11df7cffb84a38"}
Jul 14 01:43:51.240444 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:51.240370    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:52.219979 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:52.216611    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh" event={"ID":"4d997dfc-d67d-4090-8de2-1486d6c0d44d","Type":"ContainerStarted","Data":"12812e51f1f183a98c5886a26a8f6464257674221f8fcecd104090c97bb4e8fb"}
Jul 14 01:43:52.219979 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:52.217674    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"
Jul 14 01:43:52.232808 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:52.232753    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh" podStartSLOduration=1.8805017739999998 podStartE2EDuration="2.232737982s" totalImagesPullingTime="352.236208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:43:52.232654129 +0000 UTC m=+2661.797063185" watchObservedRunningTime="2026-07-14 01:43:52.232737982 +0000 UTC m=+2661.797147052"
Jul 14 01:43:54.973030 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:54.972991    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:43:55.236898 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:55.236812    2567 generic.go:356] "Generic (PLEG): container finished" podID="1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" containerID="a28a19d14984dcadeaf7621a3768018105a659feee19b7e280f6120c21df79eb" exitCode=0
Jul 14 01:43:55.236898 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:55.236857    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq" event={"ID":"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123","Type":"ContainerDied","Data":"a28a19d14984dcadeaf7621a3768018105a659feee19b7e280f6120c21df79eb"}
Jul 14 01:43:55.237256 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:55.237243    2567 scope.go:117] "RemoveContainer" containerID="a28a19d14984dcadeaf7621a3768018105a659feee19b7e280f6120c21df79eb"
Jul 14 01:43:55.238228 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:55.238198    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:43:55.238350 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:55.238298    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:43:55.238419 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:55.238352    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 14 01:43:56.147362 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:56.147318    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:56.248735 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:56.248693    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq" event={"ID":"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123","Type":"ContainerStarted","Data":"9f1e1429a83706fb787b8ddc5b1726af9277bb0b13b2434751981fc525bc027b"}
Jul 14 01:43:56.752738 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:56.752691    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:43:56.752979 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:56.752863    2567 patch_prober.go:28] interesting pod/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 14 01:43:56.752979 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:56.752912    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq" podUID="1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 14 01:43:57.752237 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:57.752197    2567 patch_prober.go:28] interesting pod/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 14 01:43:57.752866 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:57.752828    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq" podUID="1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 14 01:43:58.229361 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:58.229332    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"
Jul 14 01:43:58.752279 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:58.752239    2567 patch_prober.go:28] interesting pod/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 14 01:43:58.752930 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:58.752895    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq" podUID="1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 14 01:43:59.751503 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:59.751462    2567 patch_prober.go:28] interesting pod/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 14 01:43:59.751822 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:43:59.751791    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq" podUID="1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 14 01:44:00.755553 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:44:00.755521    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:44:00.756052 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:44:00.755799    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:44:00.756738 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:44:00.756718    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:44:03.586288 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:44:03.586256    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:44:04.817030 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:44:04.816986    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:44:09.362018 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:44:09.361979    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:44:09.460522 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:44:09.460489    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:44:10.047167 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:44:10.047124    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:44:11.853529 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:44:11.853495    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:44:11.955394 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:44:11.955357    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:44:41.678111 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:44:41.678073    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:45:34.874268 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:45:34.874232    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:45:45.262669 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:45:45.262633    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:46:26.629057 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:26.629008    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:46:26.729667 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:26.729635    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:46:40.253325 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:40.253296    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:46:41.074838 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:41.074800    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:46:54.744059 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.744019    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"]
Jul 14 01:46:54.747670 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.747642    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.750670 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.750644    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99e8d0\"/\"sub-test-model-99e8d0-kserve-self-signed-certs\""
Jul 14 01:46:54.750819 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.750724    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-99e8d0\"/\"openshift-service-ca.crt\""
Jul 14 01:46:54.750819 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.750740    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99e8d0\"/\"default-dockercfg-xlj5p\""
Jul 14 01:46:54.750819 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.750790    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-99e8d0\"/\"kube-root-ca.crt\""
Jul 14 01:46:54.759385 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.759360    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"]
Jul 14 01:46:54.802354 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.802317    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:46:54.850244 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.850206    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-dshm\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.850244 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.850248    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-tmp-dir\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.850516 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.850269    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q48jt\" (UniqueName: \"kubernetes.io/projected/a786052a-b50a-401e-87c4-908da4f98573-kube-api-access-q48jt\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.850516 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.850341    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-home\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.850516 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.850437    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-model-cache\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.850516 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.850471    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a786052a-b50a-401e-87c4-908da4f98573-tls-certs\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.951575 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.951520    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-model-cache\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.951575 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.951573    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a786052a-b50a-401e-87c4-908da4f98573-tls-certs\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.951872 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.951610    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-dshm\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.951872 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.951628    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-tmp-dir\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.951872 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.951647    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q48jt\" (UniqueName: \"kubernetes.io/projected/a786052a-b50a-401e-87c4-908da4f98573-kube-api-access-q48jt\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.951872 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.951696    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-home\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.952117 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.952060    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-model-cache\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.952180 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.952111    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-tmp-dir\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.952242 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.952204    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-home\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.954188 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.954163    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-dshm\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.954452 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.954433    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a786052a-b50a-401e-87c4-908da4f98573-tls-certs\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:54.958749 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:54.958725    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q48jt\" (UniqueName: \"kubernetes.io/projected/a786052a-b50a-401e-87c4-908da4f98573-kube-api-access-q48jt\") pod \"sub-test-model-99e8d0-kserve-9fcff64f-mww67\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") " pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:55.059145 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:55.059060    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:46:55.195002 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:55.194975    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"]
Jul 14 01:46:55.197048 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:46:55.197012    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda786052a_b50a_401e_87c4_908da4f98573.slice/crio-b8c90dda640a40e32ab4826060c93fa98a74b993c0058e5c76dd694bfe758144 WatchSource:0}: Error finding container b8c90dda640a40e32ab4826060c93fa98a74b993c0058e5c76dd694bfe758144: Status 404 returned error can't find the container with id b8c90dda640a40e32ab4826060c93fa98a74b993c0058e5c76dd694bfe758144
Jul 14 01:46:55.233032 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:55.232993    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67" event={"ID":"a786052a-b50a-401e-87c4-908da4f98573","Type":"ContainerStarted","Data":"b8c90dda640a40e32ab4826060c93fa98a74b993c0058e5c76dd694bfe758144"}
Jul 14 01:46:56.239421 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:56.239389    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67" event={"ID":"a786052a-b50a-401e-87c4-908da4f98573","Type":"ContainerStarted","Data":"ebc55fbf0e1dd6dc0fe780a8efe4228298cf8eafb4e7153ac144b0f79a713f4a"}
Jul 14 01:46:56.256459 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:46:56.256407    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67" podStartSLOduration=2.256389515 podStartE2EDuration="2.256389515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:46:56.253887228 +0000 UTC m=+2845.818296282" watchObservedRunningTime="2026-07-14 01:46:56.256389515 +0000 UTC m=+2845.820798570"
Jul 14 01:47:05.059859 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:47:05.059804    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:47:05.060523 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:47:05.060161    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:47:05.072629 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:47:05.072600    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:47:05.299081 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:47:05.299048    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:47:28.388129 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:47:28.388044    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:47:54.628385 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:47:54.628347    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:48:18.826188 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:18.826153    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"]
Jul 14 01:48:18.883126 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:18.883096    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:48:19.843620 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:19.843581    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:48:27.075850 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:27.075802    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:48:33.847334 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:33.847288    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67" podUID="a786052a-b50a-401e-87c4-908da4f98573" containerName="main" containerID="cri-o://ebc55fbf0e1dd6dc0fe780a8efe4228298cf8eafb4e7153ac144b0f79a713f4a" gracePeriod=45
Jul 14 01:48:34.106754 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.106686    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:48:34.132275 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.132246    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-home\") pod \"a786052a-b50a-401e-87c4-908da4f98573\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") "
Jul 14 01:48:34.132445 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.132291    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a786052a-b50a-401e-87c4-908da4f98573-tls-certs\") pod \"a786052a-b50a-401e-87c4-908da4f98573\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") "
Jul 14 01:48:34.132445 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.132312    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-dshm\") pod \"a786052a-b50a-401e-87c4-908da4f98573\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") "
Jul 14 01:48:34.132445 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.132347    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-model-cache\") pod \"a786052a-b50a-401e-87c4-908da4f98573\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") "
Jul 14 01:48:34.132445 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.132382    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-tmp-dir\") pod \"a786052a-b50a-401e-87c4-908da4f98573\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") "
Jul 14 01:48:34.132665 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.132475    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q48jt\" (UniqueName: \"kubernetes.io/projected/a786052a-b50a-401e-87c4-908da4f98573-kube-api-access-q48jt\") pod \"a786052a-b50a-401e-87c4-908da4f98573\" (UID: \"a786052a-b50a-401e-87c4-908da4f98573\") "
Jul 14 01:48:34.132665 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.132537    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-home" (OuterVolumeSpecName: "home") pod "a786052a-b50a-401e-87c4-908da4f98573" (UID: "a786052a-b50a-401e-87c4-908da4f98573"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:48:34.132665 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.132642    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-model-cache" (OuterVolumeSpecName: "model-cache") pod "a786052a-b50a-401e-87c4-908da4f98573" (UID: "a786052a-b50a-401e-87c4-908da4f98573"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:48:34.132816 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.132690    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a786052a-b50a-401e-87c4-908da4f98573" (UID: "a786052a-b50a-401e-87c4-908da4f98573"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:48:34.132878 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.132835    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:48:34.132878 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.132853    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:48:34.132878 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.132866    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:48:34.134779 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.134750    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a786052a-b50a-401e-87c4-908da4f98573-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a786052a-b50a-401e-87c4-908da4f98573" (UID: "a786052a-b50a-401e-87c4-908da4f98573"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:48:34.134933 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.134829    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a786052a-b50a-401e-87c4-908da4f98573-kube-api-access-q48jt" (OuterVolumeSpecName: "kube-api-access-q48jt") pod "a786052a-b50a-401e-87c4-908da4f98573" (UID: "a786052a-b50a-401e-87c4-908da4f98573"). InnerVolumeSpecName "kube-api-access-q48jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:48:34.134933 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.134834    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-dshm" (OuterVolumeSpecName: "dshm") pod "a786052a-b50a-401e-87c4-908da4f98573" (UID: "a786052a-b50a-401e-87c4-908da4f98573"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:48:34.233508 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.233471    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a786052a-b50a-401e-87c4-908da4f98573-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:48:34.233508 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.233502    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a786052a-b50a-401e-87c4-908da4f98573-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:48:34.233508 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.233512    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q48jt\" (UniqueName: \"kubernetes.io/projected/a786052a-b50a-401e-87c4-908da4f98573-kube-api-access-q48jt\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:48:34.784843 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.784803    2567 generic.go:356] "Generic (PLEG): container finished" podID="a786052a-b50a-401e-87c4-908da4f98573" containerID="ebc55fbf0e1dd6dc0fe780a8efe4228298cf8eafb4e7153ac144b0f79a713f4a" exitCode=0
Jul 14 01:48:34.785041 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.784850    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"
Jul 14 01:48:34.785041 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.784889    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67" event={"ID":"a786052a-b50a-401e-87c4-908da4f98573","Type":"ContainerDied","Data":"ebc55fbf0e1dd6dc0fe780a8efe4228298cf8eafb4e7153ac144b0f79a713f4a"}
Jul 14 01:48:34.785041 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.784934    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67" event={"ID":"a786052a-b50a-401e-87c4-908da4f98573","Type":"ContainerDied","Data":"b8c90dda640a40e32ab4826060c93fa98a74b993c0058e5c76dd694bfe758144"}
Jul 14 01:48:34.785041 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.784966    2567 scope.go:117] "RemoveContainer" containerID="ebc55fbf0e1dd6dc0fe780a8efe4228298cf8eafb4e7153ac144b0f79a713f4a"
Jul 14 01:48:34.794583 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.794559    2567 scope.go:117] "RemoveContainer" containerID="ebc55fbf0e1dd6dc0fe780a8efe4228298cf8eafb4e7153ac144b0f79a713f4a"
Jul 14 01:48:34.794886 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:48:34.794867    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc55fbf0e1dd6dc0fe780a8efe4228298cf8eafb4e7153ac144b0f79a713f4a\": container with ID starting with ebc55fbf0e1dd6dc0fe780a8efe4228298cf8eafb4e7153ac144b0f79a713f4a not found: ID does not exist" containerID="ebc55fbf0e1dd6dc0fe780a8efe4228298cf8eafb4e7153ac144b0f79a713f4a"
Jul 14 01:48:34.794944 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.794895    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc55fbf0e1dd6dc0fe780a8efe4228298cf8eafb4e7153ac144b0f79a713f4a"} err="failed to get container status \"ebc55fbf0e1dd6dc0fe780a8efe4228298cf8eafb4e7153ac144b0f79a713f4a\": rpc error: code = NotFound desc = could not find container \"ebc55fbf0e1dd6dc0fe780a8efe4228298cf8eafb4e7153ac144b0f79a713f4a\": container with ID starting with ebc55fbf0e1dd6dc0fe780a8efe4228298cf8eafb4e7153ac144b0f79a713f4a not found: ID does not exist"
Jul 14 01:48:34.806877 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.806842    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"]
Jul 14 01:48:34.810697 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.810665    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-99e8d0/sub-test-model-99e8d0-kserve-9fcff64f-mww67"]
Jul 14 01:48:34.975019 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:34.974979    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a786052a-b50a-401e-87c4-908da4f98573" path="/var/lib/kubelet/pods/a786052a-b50a-401e-87c4-908da4f98573/volumes"
Jul 14 01:48:57.571475 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:48:57.571438    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:49:04.443143 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:49:04.443109    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:49:04.543345 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:49:04.543281    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:49:12.673516 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:49:12.673480    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:49:13.857490 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:49:13.857445    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:49:43.867653 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:49:43.867622    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:49:44.745458 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:49:44.745424    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:04.403391 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:04.403355    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:04.503699 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:04.503665    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:14.447547 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:14.447506    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"]
Jul 14 01:50:14.448075 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:14.447809    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd" podUID="a27e93a0-3953-4d08-9fe7-e73f09b886b3" containerName="maas-api" containerID="cri-o://01dcdb6de01fcc857482e9da9bdd6cf389e0a38502b5e3bc2807debee8945982" gracePeriod=30
Jul 14 01:50:14.470407 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:14.470376    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:14.714497 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:14.714470    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"
Jul 14 01:50:14.722050 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:14.722009    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqkz2\" (UniqueName: \"kubernetes.io/projected/a27e93a0-3953-4d08-9fe7-e73f09b886b3-kube-api-access-kqkz2\") pod \"a27e93a0-3953-4d08-9fe7-e73f09b886b3\" (UID: \"a27e93a0-3953-4d08-9fe7-e73f09b886b3\") "
Jul 14 01:50:14.722244 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:14.722123    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a27e93a0-3953-4d08-9fe7-e73f09b886b3-maas-api-tls\") pod \"a27e93a0-3953-4d08-9fe7-e73f09b886b3\" (UID: \"a27e93a0-3953-4d08-9fe7-e73f09b886b3\") "
Jul 14 01:50:14.722244 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:14.722173    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27e93a0-3953-4d08-9fe7-e73f09b886b3-openshift-service-ca\") pod \"a27e93a0-3953-4d08-9fe7-e73f09b886b3\" (UID: \"a27e93a0-3953-4d08-9fe7-e73f09b886b3\") "
Jul 14 01:50:14.722567 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:14.722543    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27e93a0-3953-4d08-9fe7-e73f09b886b3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a27e93a0-3953-4d08-9fe7-e73f09b886b3" (UID: "a27e93a0-3953-4d08-9fe7-e73f09b886b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:50:14.724428 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:14.724401    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a27e93a0-3953-4d08-9fe7-e73f09b886b3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a27e93a0-3953-4d08-9fe7-e73f09b886b3" (UID: "a27e93a0-3953-4d08-9fe7-e73f09b886b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:50:14.724428 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:14.724409    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27e93a0-3953-4d08-9fe7-e73f09b886b3-kube-api-access-kqkz2" (OuterVolumeSpecName: "kube-api-access-kqkz2") pod "a27e93a0-3953-4d08-9fe7-e73f09b886b3" (UID: "a27e93a0-3953-4d08-9fe7-e73f09b886b3"). InnerVolumeSpecName "kube-api-access-kqkz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:50:14.822876 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:14.822825    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27e93a0-3953-4d08-9fe7-e73f09b886b3-openshift-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:50:14.822876 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:14.822869    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqkz2\" (UniqueName: \"kubernetes.io/projected/a27e93a0-3953-4d08-9fe7-e73f09b886b3-kube-api-access-kqkz2\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:50:14.822876 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:14.822880    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a27e93a0-3953-4d08-9fe7-e73f09b886b3-maas-api-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:50:15.341998 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:15.341891    2567 generic.go:356] "Generic (PLEG): container finished" podID="a27e93a0-3953-4d08-9fe7-e73f09b886b3" containerID="01dcdb6de01fcc857482e9da9bdd6cf389e0a38502b5e3bc2807debee8945982" exitCode=0
Jul 14 01:50:15.341998 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:15.341943    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"
Jul 14 01:50:15.341998 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:15.341977    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd" event={"ID":"a27e93a0-3953-4d08-9fe7-e73f09b886b3","Type":"ContainerDied","Data":"01dcdb6de01fcc857482e9da9bdd6cf389e0a38502b5e3bc2807debee8945982"}
Jul 14 01:50:15.342267 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:15.342035    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd" event={"ID":"a27e93a0-3953-4d08-9fe7-e73f09b886b3","Type":"ContainerDied","Data":"4db2152cde851b46a2bd2574cb9c03938527f60e6234a8635845f9304e002652"}
Jul 14 01:50:15.342267 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:15.342057    2567 scope.go:117] "RemoveContainer" containerID="01dcdb6de01fcc857482e9da9bdd6cf389e0a38502b5e3bc2807debee8945982"
Jul 14 01:50:15.351516 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:15.351489    2567 scope.go:117] "RemoveContainer" containerID="01dcdb6de01fcc857482e9da9bdd6cf389e0a38502b5e3bc2807debee8945982"
Jul 14 01:50:15.351812 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:50:15.351788    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01dcdb6de01fcc857482e9da9bdd6cf389e0a38502b5e3bc2807debee8945982\": container with ID starting with 01dcdb6de01fcc857482e9da9bdd6cf389e0a38502b5e3bc2807debee8945982 not found: ID does not exist" containerID="01dcdb6de01fcc857482e9da9bdd6cf389e0a38502b5e3bc2807debee8945982"
Jul 14 01:50:15.351871 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:15.351830    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01dcdb6de01fcc857482e9da9bdd6cf389e0a38502b5e3bc2807debee8945982"} err="failed to get container status \"01dcdb6de01fcc857482e9da9bdd6cf389e0a38502b5e3bc2807debee8945982\": rpc error: code = NotFound desc = could not find container \"01dcdb6de01fcc857482e9da9bdd6cf389e0a38502b5e3bc2807debee8945982\": container with ID starting with 01dcdb6de01fcc857482e9da9bdd6cf389e0a38502b5e3bc2807debee8945982 not found: ID does not exist"
Jul 14 01:50:15.356897 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:15.356870    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"]
Jul 14 01:50:15.360595 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:15.360571    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99e8d0-8878c5d65-dq8cd"]
Jul 14 01:50:16.974871 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:16.974831    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a27e93a0-3953-4d08-9fe7-e73f09b886b3" path="/var/lib/kubelet/pods/a27e93a0-3953-4d08-9fe7-e73f09b886b3/volumes"
Jul 14 01:50:18.988233 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:18.988195    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:24.899010 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:24.898901    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:24.999460 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:24.999425    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:27.823890 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:27.823850    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:29.695230 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:29.695192    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"]
Jul 14 01:50:29.695761 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:29.695540    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq" podUID="1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" containerName="istio-proxy" containerID="cri-o://9f1e1429a83706fb787b8ddc5b1726af9277bb0b13b2434751981fc525bc027b" gracePeriod=30
Jul 14 01:50:29.712049 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:29.712006    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:29.955471 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:29.955365    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:30.634612 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:30.634570    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:30.736888 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:30.736841    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:30.758527 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:30.758484    2567 patch_prober.go:28] interesting pod/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 01:50:30.758704 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:30.758539    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq" podUID="1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 01:50:34.943176 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.943145    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:50:34.971004 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.970897    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-envoy\") pod \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") "
Jul 14 01:50:34.971004 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.970997    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-workload-certs\") pod \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") "
Jul 14 01:50:34.971226 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.971030    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istiod-ca-cert\") pod \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") "
Jul 14 01:50:34.971226 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.971097    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-podinfo\") pod \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") "
Jul 14 01:50:34.971226 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.971123    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-token\") pod \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") "
Jul 14 01:50:34.971226 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.971152    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-credential-socket\") pod \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") "
Jul 14 01:50:34.971226 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.971201    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5l4z\" (UniqueName: \"kubernetes.io/projected/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-kube-api-access-l5l4z\") pod \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") "
Jul 14 01:50:34.971516 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.971231    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-workload-socket\") pod \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") "
Jul 14 01:50:34.971516 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.971269    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-data\") pod \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\" (UID: \"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123\") "
Jul 14 01:50:34.971516 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.971347    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" (UID: "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:50:34.971679 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.971567    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-workload-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:50:34.971679 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.971587    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" (UID: "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:50:34.971807 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.971775    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" (UID: "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:50:34.972060 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.972029    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" (UID: "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:50:34.972734 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.972704    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-data" (OuterVolumeSpecName: "istio-data") pod "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" (UID: "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:50:34.973814 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.973774    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-kube-api-access-l5l4z" (OuterVolumeSpecName: "kube-api-access-l5l4z") pod "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" (UID: "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123"). InnerVolumeSpecName "kube-api-access-l5l4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:50:34.973920 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.973822    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" (UID: "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:50:34.974177 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.974154    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-token" (OuterVolumeSpecName: "istio-token") pod "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" (UID: "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:50:34.974365 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:34.974348    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" (UID: "1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 01:50:35.072950 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.072912    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-podinfo\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:50:35.072950 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.072944    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-token\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:50:35.072950 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.072980    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-credential-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:50:35.072950 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.072990    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5l4z\" (UniqueName: \"kubernetes.io/projected/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-kube-api-access-l5l4z\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:50:35.073272 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.072999    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-workload-socket\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:50:35.073272 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.073009    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-data\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:50:35.073272 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.073017    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istio-envoy\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:50:35.073272 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.073025    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123-istiod-ca-cert\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:50:35.448713 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.448677    2567 generic.go:356] "Generic (PLEG): container finished" podID="1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" containerID="9f1e1429a83706fb787b8ddc5b1726af9277bb0b13b2434751981fc525bc027b" exitCode=0
Jul 14 01:50:35.448902 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.448726    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq" event={"ID":"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123","Type":"ContainerDied","Data":"9f1e1429a83706fb787b8ddc5b1726af9277bb0b13b2434751981fc525bc027b"}
Jul 14 01:50:35.448902 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.448771    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq" event={"ID":"1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123","Type":"ContainerDied","Data":"e35e3ded4d725833ebafc71673cda7969d6f81503ee825378938adb72bc003f6"}
Jul 14 01:50:35.448902 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.448796    2567 scope.go:117] "RemoveContainer" containerID="9f1e1429a83706fb787b8ddc5b1726af9277bb0b13b2434751981fc525bc027b"
Jul 14 01:50:35.448902 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.448730    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"
Jul 14 01:50:35.458134 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.458111    2567 scope.go:117] "RemoveContainer" containerID="a28a19d14984dcadeaf7621a3768018105a659feee19b7e280f6120c21df79eb"
Jul 14 01:50:35.467525 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.467452    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"]
Jul 14 01:50:35.468879 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.468861    2567 scope.go:117] "RemoveContainer" containerID="9f1e1429a83706fb787b8ddc5b1726af9277bb0b13b2434751981fc525bc027b"
Jul 14 01:50:35.469215 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:50:35.469195    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1e1429a83706fb787b8ddc5b1726af9277bb0b13b2434751981fc525bc027b\": container with ID starting with 9f1e1429a83706fb787b8ddc5b1726af9277bb0b13b2434751981fc525bc027b not found: ID does not exist" containerID="9f1e1429a83706fb787b8ddc5b1726af9277bb0b13b2434751981fc525bc027b"
Jul 14 01:50:35.469282 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.469225    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1e1429a83706fb787b8ddc5b1726af9277bb0b13b2434751981fc525bc027b"} err="failed to get container status \"9f1e1429a83706fb787b8ddc5b1726af9277bb0b13b2434751981fc525bc027b\": rpc error: code = NotFound desc = could not find container \"9f1e1429a83706fb787b8ddc5b1726af9277bb0b13b2434751981fc525bc027b\": container with ID starting with 9f1e1429a83706fb787b8ddc5b1726af9277bb0b13b2434751981fc525bc027b not found: ID does not exist"
Jul 14 01:50:35.469282 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.469243    2567 scope.go:117] "RemoveContainer" containerID="a28a19d14984dcadeaf7621a3768018105a659feee19b7e280f6120c21df79eb"
Jul 14 01:50:35.469456 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:50:35.469440    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28a19d14984dcadeaf7621a3768018105a659feee19b7e280f6120c21df79eb\": container with ID starting with a28a19d14984dcadeaf7621a3768018105a659feee19b7e280f6120c21df79eb not found: ID does not exist" containerID="a28a19d14984dcadeaf7621a3768018105a659feee19b7e280f6120c21df79eb"
Jul 14 01:50:35.469506 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.469460    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28a19d14984dcadeaf7621a3768018105a659feee19b7e280f6120c21df79eb"} err="failed to get container status \"a28a19d14984dcadeaf7621a3768018105a659feee19b7e280f6120c21df79eb\": rpc error: code = NotFound desc = could not find container \"a28a19d14984dcadeaf7621a3768018105a659feee19b7e280f6120c21df79eb\": container with ID starting with a28a19d14984dcadeaf7621a3768018105a659feee19b7e280f6120c21df79eb not found: ID does not exist"
Jul 14 01:50:35.470946 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:35.470922    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-99e8d0-openshift-default-85bb99b599-7z7dq"]
Jul 14 01:50:36.974732 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:36.974696    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" path="/var/lib/kubelet/pods/1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123/volumes"
Jul 14 01:50:40.646681 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:40.646639    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"]
Jul 14 01:50:40.647148 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:40.646923    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh" podUID="4d997dfc-d67d-4090-8de2-1486d6c0d44d" containerName="maas-api" containerID="cri-o://12812e51f1f183a98c5886a26a8f6464257674221f8fcecd104090c97bb4e8fb" gracePeriod=30
Jul 14 01:50:40.680481 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:40.680441    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:41.103023 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.102996    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"
Jul 14 01:50:41.121024 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.120986    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8v6l\" (UniqueName: \"kubernetes.io/projected/4d997dfc-d67d-4090-8de2-1486d6c0d44d-kube-api-access-m8v6l\") pod \"4d997dfc-d67d-4090-8de2-1486d6c0d44d\" (UID: \"4d997dfc-d67d-4090-8de2-1486d6c0d44d\") "
Jul 14 01:50:41.121275 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.121032    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d997dfc-d67d-4090-8de2-1486d6c0d44d-openshift-service-ca\") pod \"4d997dfc-d67d-4090-8de2-1486d6c0d44d\" (UID: \"4d997dfc-d67d-4090-8de2-1486d6c0d44d\") "
Jul 14 01:50:41.121275 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.121068    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d997dfc-d67d-4090-8de2-1486d6c0d44d-maas-api-tls\") pod \"4d997dfc-d67d-4090-8de2-1486d6c0d44d\" (UID: \"4d997dfc-d67d-4090-8de2-1486d6c0d44d\") "
Jul 14 01:50:41.121727 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.121438    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d997dfc-d67d-4090-8de2-1486d6c0d44d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d997dfc-d67d-4090-8de2-1486d6c0d44d" (UID: "4d997dfc-d67d-4090-8de2-1486d6c0d44d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:50:41.123452 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.123426    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d997dfc-d67d-4090-8de2-1486d6c0d44d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d997dfc-d67d-4090-8de2-1486d6c0d44d" (UID: "4d997dfc-d67d-4090-8de2-1486d6c0d44d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:50:41.123562 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.123543    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d997dfc-d67d-4090-8de2-1486d6c0d44d-kube-api-access-m8v6l" (OuterVolumeSpecName: "kube-api-access-m8v6l") pod "4d997dfc-d67d-4090-8de2-1486d6c0d44d" (UID: "4d997dfc-d67d-4090-8de2-1486d6c0d44d"). InnerVolumeSpecName "kube-api-access-m8v6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:50:41.222079 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.221950    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8v6l\" (UniqueName: \"kubernetes.io/projected/4d997dfc-d67d-4090-8de2-1486d6c0d44d-kube-api-access-m8v6l\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:50:41.222079 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.222021    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d997dfc-d67d-4090-8de2-1486d6c0d44d-openshift-service-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:50:41.222079 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.222031    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d997dfc-d67d-4090-8de2-1486d6c0d44d-maas-api-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
Jul 14 01:50:41.483193 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.483100    2567 generic.go:356] "Generic (PLEG): container finished" podID="4d997dfc-d67d-4090-8de2-1486d6c0d44d" containerID="12812e51f1f183a98c5886a26a8f6464257674221f8fcecd104090c97bb4e8fb" exitCode=0
Jul 14 01:50:41.483193 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.483144    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh" event={"ID":"4d997dfc-d67d-4090-8de2-1486d6c0d44d","Type":"ContainerDied","Data":"12812e51f1f183a98c5886a26a8f6464257674221f8fcecd104090c97bb4e8fb"}
Jul 14 01:50:41.483193 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.483151    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"
Jul 14 01:50:41.483495 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.483195    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh" event={"ID":"4d997dfc-d67d-4090-8de2-1486d6c0d44d","Type":"ContainerDied","Data":"46ed5ee3b3b3442effddc3b6b560439612d70fe513cc25208c11df7cffb84a38"}
Jul 14 01:50:41.483495 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.483224    2567 scope.go:117] "RemoveContainer" containerID="12812e51f1f183a98c5886a26a8f6464257674221f8fcecd104090c97bb4e8fb"
Jul 14 01:50:41.493739 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.493718    2567 scope.go:117] "RemoveContainer" containerID="12812e51f1f183a98c5886a26a8f6464257674221f8fcecd104090c97bb4e8fb"
Jul 14 01:50:41.494087 ip-10-0-129-117 kubenswrapper[2567]: E0714 01:50:41.494065    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12812e51f1f183a98c5886a26a8f6464257674221f8fcecd104090c97bb4e8fb\": container with ID starting with 12812e51f1f183a98c5886a26a8f6464257674221f8fcecd104090c97bb4e8fb not found: ID does not exist" containerID="12812e51f1f183a98c5886a26a8f6464257674221f8fcecd104090c97bb4e8fb"
Jul 14 01:50:41.494176 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.494094    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12812e51f1f183a98c5886a26a8f6464257674221f8fcecd104090c97bb4e8fb"} err="failed to get container status \"12812e51f1f183a98c5886a26a8f6464257674221f8fcecd104090c97bb4e8fb\": rpc error: code = NotFound desc = could not find container \"12812e51f1f183a98c5886a26a8f6464257674221f8fcecd104090c97bb4e8fb\": container with ID starting with 12812e51f1f183a98c5886a26a8f6464257674221f8fcecd104090c97bb4e8fb not found: ID does not exist"
Jul 14 01:50:41.502900 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.502873    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"]
Jul 14 01:50:41.506574 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:41.506552    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e42ce1-7764458c6f-snzbh"]
Jul 14 01:50:42.974778 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:42.974736    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d997dfc-d67d-4090-8de2-1486d6c0d44d" path="/var/lib/kubelet/pods/4d997dfc-d67d-4090-8de2-1486d6c0d44d/volumes"
Jul 14 01:50:45.186390 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:45.186350    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:50.929356 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:50.929320    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:51.029538 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:51.029495    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:55.901118 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:55.901083    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:50:56.128046 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:50:56.128013    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:51:01.788130 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:01.788095    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-tlqk9_92831026-a30e-45e9-8d0d-ec83e8d50819/server/0.log"
Jul 14 01:51:02.016184 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:02.016150    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bc876bb4c-cpstz_305439f7-e716-4f07-b05f-de32c0814257/copy-manifests/0.log"
Jul 14 01:51:02.022399 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:02.022377    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bc876bb4c-cpstz_305439f7-e716-4f07-b05f-de32c0814257/manager/0.log"
Jul 14 01:51:02.735686 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:02.735657    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29733195-kxvd4_d93b7cd2-8875-45a2-9b55-5957fe44b7cb/cleanup/0.log"
Jul 14 01:51:03.062240 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:03.062152    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-0d59bf49-sksv4_c11d7790-d915-4b9e-94d0-8255e5ba50a8/revoke-keys/0.log"
Jul 14 01:51:03.166824 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:03.166786    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-62a42e82-fbkfn_14c59f6b-e7e1-4144-b170-1389de6a0c01/revoke-keys/0.log"
Jul 14 01:51:03.486942 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:03.486910    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-0e23d941-wbqg5_52dfa3b9-1be0-403d-9f78-348ba9b191d6/revoke-keys/0.log"
Jul 14 01:51:03.821273 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:03.821190    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-7149234b-h5dmx_3e30ff65-06f2-4a44-be6a-648d86d0fa23/revoke-keys/0.log"
Jul 14 01:51:04.040374 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:04.040346    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-db5a4a31-8v8sk_b4f9639a-6472-4942-b7e0-994bd0223824/revoke-keys/0.log"
Jul 14 01:51:05.132905 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:05.132874    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4_f519d49d-6e68-448b-b529-981875cdc8cc/util/0.log"
Jul 14 01:51:05.139126 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:05.139102    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4_f519d49d-6e68-448b-b529-981875cdc8cc/pull/0.log"
Jul 14 01:51:05.144827 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:05.144808    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4_f519d49d-6e68-448b-b529-981875cdc8cc/extract/0.log"
Jul 14 01:51:05.251057 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:05.251029    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt_fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86/util/0.log"
Jul 14 01:51:05.256803 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:05.256782    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt_fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86/pull/0.log"
Jul 14 01:51:05.262269 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:05.262251    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt_fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86/extract/0.log"
Jul 14 01:51:05.372024 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:05.371995    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88_7149f3c0-165a-4865-acaa-5e84b34bfdd3/extract/0.log"
Jul 14 01:51:05.377746 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:05.377728    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88_7149f3c0-165a-4865-acaa-5e84b34bfdd3/util/0.log"
Jul 14 01:51:05.385042 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:05.384981    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88_7149f3c0-165a-4865-acaa-5e84b34bfdd3/pull/0.log"
Jul 14 01:51:05.497593 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:05.497565    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s_eaa6228d-8835-49c5-a0ed-77b810d4890f/util/0.log"
Jul 14 01:51:05.503180 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:05.503156    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s_eaa6228d-8835-49c5-a0ed-77b810d4890f/pull/0.log"
Jul 14 01:51:05.508849 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:05.508824    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s_eaa6228d-8835-49c5-a0ed-77b810d4890f/extract/0.log"
Jul 14 01:51:06.779198 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:06.779171    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vgbd2_1ce796ea-a468-4538-9196-5fd3d87e0f15/istio-proxy/0.log"
Jul 14 01:51:06.895498 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:06.895462    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-tdz6p_34303f0d-42bf-4c06-ab86-2b87ba91a3f6/discovery/0.log"
Jul 14 01:51:07.349936 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:07.349890    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-hsj5q_b6934939-13dd-40c8-ae10-62971cf22838/payload-pre-processing/0.log"
Jul 14 01:51:07.572896 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:07.572870    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c77944dd8-qlg7c_a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b/router/0.log"
Jul 14 01:51:07.895402 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:07.895364    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-mg2b6_dffd9eb6-560e-4ae1-9e09-84319ca10461/main/0.log"
Jul 14 01:51:08.002896 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:08.002863    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-g6r55_47660f12-7c2a-455e-975d-213ebadf129b/main/0.log"
Jul 14 01:51:12.853496 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:12.853454    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:51:12.918372 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:12.918338    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:51:15.624474 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:15.624443    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q8j4f_1a747595-65ad-41fe-a1f1-d0914d6eadc2/global-pull-secret-syncer/0.log"
Jul 14 01:51:15.667544 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:15.667512    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log"
Jul 14 01:51:15.769814 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:15.769782    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-117.ec2.internal_9105b408cddaedf77cc3bc80f328aa9e/haproxy/0.log"
Jul 14 01:51:20.368804 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:20.368769    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4_f519d49d-6e68-448b-b529-981875cdc8cc/extract/0.log"
Jul 14 01:51:20.385265 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:20.385228    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4_f519d49d-6e68-448b-b529-981875cdc8cc/util/0.log"
Jul 14 01:51:20.403482 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:20.403456    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598dth4_f519d49d-6e68-448b-b529-981875cdc8cc/pull/0.log"
Jul 14 01:51:20.424770 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:20.424738    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt_fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86/extract/0.log"
Jul 14 01:51:20.440421 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:20.440391    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt_fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86/util/0.log"
Jul 14 01:51:20.455681 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:20.455653    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e06wcbt_fe84fb62-ae2f-4a3e-aba4-183d1d0f4a86/pull/0.log"
Jul 14 01:51:20.476516 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:20.476486    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88_7149f3c0-165a-4865-acaa-5e84b34bfdd3/extract/0.log"
Jul 14 01:51:20.495689 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:20.495657    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88_7149f3c0-165a-4865-acaa-5e84b34bfdd3/util/0.log"
Jul 14 01:51:20.511087 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:20.511061    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73sms88_7149f3c0-165a-4865-acaa-5e84b34bfdd3/pull/0.log"
Jul 14 01:51:20.531237 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:20.531210    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s_eaa6228d-8835-49c5-a0ed-77b810d4890f/extract/0.log"
Jul 14 01:51:20.546509 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:20.546484    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s_eaa6228d-8835-49c5-a0ed-77b810d4890f/util/0.log"
Jul 14 01:51:20.561976 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:20.561929    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jhk2s_eaa6228d-8835-49c5-a0ed-77b810d4890f/pull/0.log"
Jul 14 01:51:23.240520 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:23.240490    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-t4nt6_a5b6f961-4322-40e4-a5b1-12efb6f8e785/monitoring-plugin/0.log"
Jul 14 01:51:23.268707 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:23.268679    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4fcbl_047eff54-d982-4e1f-a152-46ffd29c8db2/node-exporter/0.log"
Jul 14 01:51:23.288554 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:23.288523    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4fcbl_047eff54-d982-4e1f-a152-46ffd29c8db2/kube-rbac-proxy/0.log"
Jul 14 01:51:23.302713 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:23.302687    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4fcbl_047eff54-d982-4e1f-a152-46ffd29c8db2/init-textfile/0.log"
Jul 14 01:51:23.694511 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:23.694477    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-k24bj_32a8e14b-c44c-48c4-90af-656a3555d54e/prometheus-operator/0.log"
Jul 14 01:51:23.708155 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:23.708127    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-k24bj_32a8e14b-c44c-48c4-90af-656a3555d54e/kube-rbac-proxy/0.log"
Jul 14 01:51:23.730891 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:23.730862    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-m4tmv_77bd9659-0c61-44e0-bc53-0f2084fa0bb6/prometheus-operator-admission-webhook/0.log"
Jul 14 01:51:25.159571 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:25.159539    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-jhd7p_66ca18b4-5345-4c49-9772-e4bfd1bc6747/networking-console-plugin/0.log"
Jul 14 01:51:26.241867 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.241827    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"]
Jul 14 01:51:26.242269 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.242172    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d997dfc-d67d-4090-8de2-1486d6c0d44d" containerName="maas-api"
Jul 14 01:51:26.242269 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.242187    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d997dfc-d67d-4090-8de2-1486d6c0d44d" containerName="maas-api"
Jul 14 01:51:26.242269 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.242206    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27e93a0-3953-4d08-9fe7-e73f09b886b3" containerName="maas-api"
Jul 14 01:51:26.242269 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.242213    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27e93a0-3953-4d08-9fe7-e73f09b886b3" containerName="maas-api"
Jul 14 01:51:26.242269 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.242219    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a786052a-b50a-401e-87c4-908da4f98573" containerName="main"
Jul 14 01:51:26.242269 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.242225    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a786052a-b50a-401e-87c4-908da4f98573" containerName="main"
Jul 14 01:51:26.242269 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.242234    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" containerName="istio-proxy"
Jul 14 01:51:26.242269 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.242240    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" containerName="istio-proxy"
Jul 14 01:51:26.242269 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.242249    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" containerName="istio-proxy"
Jul 14 01:51:26.242269 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.242254    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" containerName="istio-proxy"
Jul 14 01:51:26.242790 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.242314    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27e93a0-3953-4d08-9fe7-e73f09b886b3" containerName="maas-api"
Jul 14 01:51:26.242790 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.242323    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a786052a-b50a-401e-87c4-908da4f98573" containerName="main"
Jul 14 01:51:26.242790 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.242329    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" containerName="istio-proxy"
Jul 14 01:51:26.242790 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.242334    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cf34f8c-b7ad-4e0c-8206-ba2bd55bc123" containerName="istio-proxy"
Jul 14 01:51:26.242790 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.242339    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d997dfc-d67d-4090-8de2-1486d6c0d44d" containerName="maas-api"
Jul 14 01:51:26.245785 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.245766    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.247697 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.247674    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-njqhj\"/\"kube-root-ca.crt\""
Jul 14 01:51:26.247697 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.247674    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-njqhj\"/\"openshift-service-ca.crt\""
Jul 14 01:51:26.247878 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.247745    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-njqhj\"/\"default-dockercfg-78zpm\""
Jul 14 01:51:26.255423 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.255395    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"]
Jul 14 01:51:26.325375 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.325334    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2bd09f63-d344-4ad4-9f07-00ff29365380-proc\") pod \"perf-node-gather-daemonset-ps4kq\" (UID: \"2bd09f63-d344-4ad4-9f07-00ff29365380\") " pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.325582 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.325382    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvghj\" (UniqueName: \"kubernetes.io/projected/2bd09f63-d344-4ad4-9f07-00ff29365380-kube-api-access-qvghj\") pod \"perf-node-gather-daemonset-ps4kq\" (UID: \"2bd09f63-d344-4ad4-9f07-00ff29365380\") " pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.325582 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.325463    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2bd09f63-d344-4ad4-9f07-00ff29365380-lib-modules\") pod \"perf-node-gather-daemonset-ps4kq\" (UID: \"2bd09f63-d344-4ad4-9f07-00ff29365380\") " pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.325582 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.325483    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bd09f63-d344-4ad4-9f07-00ff29365380-sys\") pod \"perf-node-gather-daemonset-ps4kq\" (UID: \"2bd09f63-d344-4ad4-9f07-00ff29365380\") " pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.325582 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.325509    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2bd09f63-d344-4ad4-9f07-00ff29365380-podres\") pod \"perf-node-gather-daemonset-ps4kq\" (UID: \"2bd09f63-d344-4ad4-9f07-00ff29365380\") " pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.425901 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.425857    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2bd09f63-d344-4ad4-9f07-00ff29365380-lib-modules\") pod \"perf-node-gather-daemonset-ps4kq\" (UID: \"2bd09f63-d344-4ad4-9f07-00ff29365380\") " pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.425901 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.425899    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bd09f63-d344-4ad4-9f07-00ff29365380-sys\") pod \"perf-node-gather-daemonset-ps4kq\" (UID: \"2bd09f63-d344-4ad4-9f07-00ff29365380\") " pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.426143 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.425921    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2bd09f63-d344-4ad4-9f07-00ff29365380-podres\") pod \"perf-node-gather-daemonset-ps4kq\" (UID: \"2bd09f63-d344-4ad4-9f07-00ff29365380\") " pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.426143 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.426009    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2bd09f63-d344-4ad4-9f07-00ff29365380-proc\") pod \"perf-node-gather-daemonset-ps4kq\" (UID: \"2bd09f63-d344-4ad4-9f07-00ff29365380\") " pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.426143 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.426030    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2bd09f63-d344-4ad4-9f07-00ff29365380-sys\") pod \"perf-node-gather-daemonset-ps4kq\" (UID: \"2bd09f63-d344-4ad4-9f07-00ff29365380\") " pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.426143 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.426032    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvghj\" (UniqueName: \"kubernetes.io/projected/2bd09f63-d344-4ad4-9f07-00ff29365380-kube-api-access-qvghj\") pod \"perf-node-gather-daemonset-ps4kq\" (UID: \"2bd09f63-d344-4ad4-9f07-00ff29365380\") " pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.426143 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.426076    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2bd09f63-d344-4ad4-9f07-00ff29365380-lib-modules\") pod \"perf-node-gather-daemonset-ps4kq\" (UID: \"2bd09f63-d344-4ad4-9f07-00ff29365380\") " pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.426143 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.426124    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2bd09f63-d344-4ad4-9f07-00ff29365380-proc\") pod \"perf-node-gather-daemonset-ps4kq\" (UID: \"2bd09f63-d344-4ad4-9f07-00ff29365380\") " pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.426371 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.426140    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2bd09f63-d344-4ad4-9f07-00ff29365380-podres\") pod \"perf-node-gather-daemonset-ps4kq\" (UID: \"2bd09f63-d344-4ad4-9f07-00ff29365380\") " pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.433674 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.433647    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvghj\" (UniqueName: \"kubernetes.io/projected/2bd09f63-d344-4ad4-9f07-00ff29365380-kube-api-access-qvghj\") pod \"perf-node-gather-daemonset-ps4kq\" (UID: \"2bd09f63-d344-4ad4-9f07-00ff29365380\") " pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.557543 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.557452    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:26.695915 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.694616    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"]
Jul 14 01:51:26.698748 ip-10-0-129-117 kubenswrapper[2567]: W0714 01:51:26.698703    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2bd09f63_d344_4ad4_9f07_00ff29365380.slice/crio-c25f9f2ccff687519995ce2c9b97b9b01955b24f0d6ecc85b81187f18b122fb4 WatchSource:0}: Error finding container c25f9f2ccff687519995ce2c9b97b9b01955b24f0d6ecc85b81187f18b122fb4: Status 404 returned error can't find the container with id c25f9f2ccff687519995ce2c9b97b9b01955b24f0d6ecc85b81187f18b122fb4
Jul 14 01:51:26.723031 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:26.722988    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq" event={"ID":"2bd09f63-d344-4ad4-9f07-00ff29365380","Type":"ContainerStarted","Data":"c25f9f2ccff687519995ce2c9b97b9b01955b24f0d6ecc85b81187f18b122fb4"}
Jul 14 01:51:27.335200 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:27.335168    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:51:27.435544 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:27.435512    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z2tl_336250ce-5e32-48f2-88a0-365516e9b4b7/konnectivity-agent/0.log" limit=1024
Jul 14 01:51:27.729990 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:27.729935    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq" event={"ID":"2bd09f63-d344-4ad4-9f07-00ff29365380","Type":"ContainerStarted","Data":"110aaecace25342ef02df9aea9b77f52399828172f7862bafcd4e8f66f00c38c"}
Jul 14 01:51:27.730184 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:27.730130    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:27.745476 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:27.745388    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq" podStartSLOduration=1.745365855 podStartE2EDuration="1.745365855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 01:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:51:27.743982386 +0000 UTC m=+3117.308391439" watchObservedRunningTime="2026-07-14 01:51:27.745365855 +0000 UTC m=+3117.309774911"
Jul 14 01:51:27.946779 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:27.946749    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jjkx4_ef8fc61b-7241-4cec-bda8-ecfb2ed789f1/dns/0.log"
Jul 14 01:51:27.961543 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:27.961498    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jjkx4_ef8fc61b-7241-4cec-bda8-ecfb2ed789f1/kube-rbac-proxy/0.log"
Jul 14 01:51:28.017530 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:28.017438    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d4fkp_2998116f-cba6-4d84-a68c-c4da115286ad/dns-node-resolver/0.log"
Jul 14 01:51:28.570245 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:28.570214    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jpnj7_b5459bba-8c2e-4be1-802f-a462a23327f6/node-ca/0.log"
Jul 14 01:51:29.438508 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:29.438474    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vgbd2_1ce796ea-a468-4538-9196-5fd3d87e0f15/istio-proxy/0.log"
Jul 14 01:51:29.554619 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:29.554574    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-tdz6p_34303f0d-42bf-4c06-ab86-2b87ba91a3f6/discovery/0.log"
Jul 14 01:51:29.709146 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:29.709050    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-hsj5q_b6934939-13dd-40c8-ae10-62971cf22838/payload-pre-processing/0.log"
Jul 14 01:51:29.770277 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:29.770238    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c77944dd8-qlg7c_a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b/router/0.log"
Jul 14 01:51:30.463056 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:30.463015    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ghmc7_c8f406c4-b03c-48ff-a9d0-d12af1c70dff/serve-healthcheck-canary/0.log"
Jul 14 01:51:31.002305 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:31.002271    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hws8z_2c4e0e77-c547-406c-b68e-0bfff407fcf2/kube-rbac-proxy/0.log"
Jul 14 01:51:31.020466 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:31.020433    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hws8z_2c4e0e77-c547-406c-b68e-0bfff407fcf2/exporter/0.log"
Jul 14 01:51:31.035927 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:31.035898    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hws8z_2c4e0e77-c547-406c-b68e-0bfff407fcf2/extractor/0.log"
Jul 14 01:51:33.561703 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:33.561671    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-tlqk9_92831026-a30e-45e9-8d0d-ec83e8d50819/server/0.log"
Jul 14 01:51:33.695979 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:33.695925    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bc876bb4c-cpstz_305439f7-e716-4f07-b05f-de32c0814257/manager/0.log"
Jul 14 01:51:33.711266 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:33.711240    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bc876bb4c-cpstz_305439f7-e716-4f07-b05f-de32c0814257/copy-manifests/0.log"
Jul 14 01:51:33.752606 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:33.752575    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-njqhj/perf-node-gather-daemonset-ps4kq"
Jul 14 01:51:35.113126 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:35.113031    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-78f77bcdb4-9l448_b3669ca4-4d0f-4bbf-b811-fd26049c94a5/manager/0.log"
Jul 14 01:51:36.865464 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:36.865435    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vgbd2_1ce796ea-a468-4538-9196-5fd3d87e0f15/istio-proxy/0.log"
Jul 14 01:51:36.971972 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:36.971927    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-tdz6p_34303f0d-42bf-4c06-ab86-2b87ba91a3f6/discovery/0.log"
Jul 14 01:51:37.097552 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:37.097519    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-hsj5q_b6934939-13dd-40c8-ae10-62971cf22838/payload-pre-processing/0.log"
Jul 14 01:51:37.140325 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:37.140297    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c77944dd8-qlg7c_a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b/router/0.log"
Jul 14 01:51:38.477064 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:38.477029    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vgbd2_1ce796ea-a468-4538-9196-5fd3d87e0f15/istio-proxy/0.log"
Jul 14 01:51:38.590894 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:38.590854    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-tdz6p_34303f0d-42bf-4c06-ab86-2b87ba91a3f6/discovery/0.log"
Jul 14 01:51:38.716610 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:38.716578    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-hsj5q_b6934939-13dd-40c8-ae10-62971cf22838/payload-pre-processing/0.log"
Jul 14 01:51:38.761171 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:38.761097    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6c77944dd8-qlg7c_a9cab32d-5e49-4a6e-b5e4-f5bbfa7e7e4b/router/0.log"
Jul 14 01:51:40.302444 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:40.302416    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-ljjxb_6e840f36-bcc1-473f-8f2d-36de1ebc376c/migrator/0.log"
Jul 14 01:51:40.319658 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:40.319625    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-ljjxb_6e840f36-bcc1-473f-8f2d-36de1ebc376c/graceful-termination/0.log"
Jul 14 01:51:41.632566 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:41.632530    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2mx5v_3a144e7e-6d83-49a5-90c8-6052b6734d80/kube-multus/0.log"
Jul 14 01:51:41.652083 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:41.652056    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cg7w8_31b7f78f-0db2-469f-886d-a83f025c6a62/kube-multus-additional-cni-plugins/0.log"
Jul 14 01:51:41.667194 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:41.667160    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cg7w8_31b7f78f-0db2-469f-886d-a83f025c6a62/egress-router-binary-copy/0.log"
Jul 14 01:51:41.682605 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:41.682580    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cg7w8_31b7f78f-0db2-469f-886d-a83f025c6a62/cni-plugins/0.log"
Jul 14 01:51:41.698369 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:41.698342    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cg7w8_31b7f78f-0db2-469f-886d-a83f025c6a62/bond-cni-plugin/0.log"
Jul 14 01:51:41.713829 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:41.713800    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cg7w8_31b7f78f-0db2-469f-886d-a83f025c6a62/routeoverride-cni/0.log"
Jul 14 01:51:41.729687 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:41.729660    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cg7w8_31b7f78f-0db2-469f-886d-a83f025c6a62/whereabouts-cni-bincopy/0.log"
Jul 14 01:51:41.748789 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:41.748763    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cg7w8_31b7f78f-0db2-469f-886d-a83f025c6a62/whereabouts-cni/0.log"
Jul 14 01:51:42.107914 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:42.107814    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jfg2f_f0ebac4e-8f5b-4820-b018-8386d03067f1/network-metrics-daemon/0.log"
Jul 14 01:51:42.124101 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:42.124070    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jfg2f_f0ebac4e-8f5b-4820-b018-8386d03067f1/kube-rbac-proxy/0.log"
Jul 14 01:51:42.877183 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:42.877154    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6kbd_ca6d2ac7-b806-485c-b968-f1e2bb8a1a45/ovn-controller/0.log"
Jul 14 01:51:42.889539 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:42.889510    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6kbd_ca6d2ac7-b806-485c-b968-f1e2bb8a1a45/ovn-acl-logging/0.log"
Jul 14 01:51:42.909109 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:42.909075    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6kbd_ca6d2ac7-b806-485c-b968-f1e2bb8a1a45/kube-rbac-proxy-node/0.log"
Jul 14 01:51:42.924812 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:42.924782    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6kbd_ca6d2ac7-b806-485c-b968-f1e2bb8a1a45/kube-rbac-proxy-ovn-metrics/0.log"
Jul 14 01:51:42.937057 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:42.937030    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6kbd_ca6d2ac7-b806-485c-b968-f1e2bb8a1a45/northd/0.log"
Jul 14 01:51:42.954385 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:42.954362    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6kbd_ca6d2ac7-b806-485c-b968-f1e2bb8a1a45/nbdb/0.log"
Jul 14 01:51:42.971051 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:42.971024    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6kbd_ca6d2ac7-b806-485c-b968-f1e2bb8a1a45/sbdb/0.log"
Jul 14 01:51:43.020288 ip-10-0-129-117 kubenswrapper[2567]: I0714 01:51:43.020256    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6kbd_ca6d2ac7-b806-485c-b968-f1e2bb8a1a45/ovnkube-controller/0.log"